Appendix
B

A Test Suite for
Programming by Demonstration

Richard Potter
and
David Maulsby

Introduction

This Appendix is the start of what should grow into a valuable resource for PBD research. The suite of example tasks that follows should help the researcher in two ways. One is to illustrate (by example) the generality of the PBD vision by showing broad practical applications across many task domains. The other is to push the generality of PBD research by allowing researchers to challenge their systems with tasks from other peoples' experiences. As tasks are contributed to the suite and as systems are compared in their ability to automate these tasks, researchers should be able to formulate a robust list of generic PBD capabilities. Classic problems needing special research attention should emerge as well.

Showing the broad applicability of PBD

The first reason for the PBD Test Suite is to show the broad applicability of PBD. There is no better way to show the "real world" usefulness of an idea than to give a specific, realistic example. Therefore many of the examples in this Appendix are realistic situations where PBD could be useful. The examples also come from many task domains and reinforce the fact that PBD is not limited to any particular task domain. PBD is not specifically about text editing, graphic editing, or file management. It is about extending the functionality of a computer system.

If each example contained only a task description, the wide variety of examples might incorrectly imply that PBD can do everything. PBD, however, does not accomplish a task in isolation. PBD relies on the base functionality of the computer system and automates a task by extending the base functionality. This relationship can be graphically depicted as in the figure. Changing either endpoint in this depiction changes what is required of the PBD system. Therefore, each example includes a detailed description of a realistic base functionality so that the amount PBD must extend the computer's existing functionality can be understood to be reasonable.


PBD extends the functionality of existing applications to accomplish new tasks. The contribution of PBD must be assessed relative to base functionality, which differs from one application program to another, as shown.

Another reason that a realistic base functionality is explicitly stated is so PBD's contribution is not overlooked. Many of the example tasks can be automated without PBD in other situations. For instance, the "Odd/even pages" task, which appears later in this chapter, can be automated with a built-in feature of MacWrite II. Perhaps a future version of the Macintosh operating system will make this feature accessible to all applications. Explicitly stating a realistic base functionality documents that a realistic situation exists where creating the feature with PBD would be desirable.

One dilemma is that the base functionality of a realistic situation is often too rich to be listed in its entirety. For example, just listing all the available ways to move the cursor (character forward/backward, word forwards/backward, beginning of line, end of line, etc.) in a realistic word processor could take several pages of explanation. Yet this detail is important because a PBD system may rely on the existence of such functionality. This chapter takes a compromise solution which is to state a well-known application where the need for the new automation can be experienced firsthand. Since no particular application is likely to be familiar to all readers (or even to more than a few readers after a number of years), the functionality most likely to be of use is also listed. This should give most readers an idea of what is required of the PBD system, and perhaps allow them to see an analogous example in their own word processor, drawing program, or other application. Readers who want more detail still have the option of seeking out the actual system and exploring the challenge of automating the task firsthand.

Pushing the generality of PBD

The second use for the PBD Test Suite is to push the generality of PBD research. Generality is important because PBD technology is meant to allow users to automate their unique repetitive tasks that could not have been predicted by application designers. PBD systems must be general to handle the unexpected. Testing their systems against tasks from the suite will help researchers discover the merits and limitations of different PBD techniques.

To ensure generality, some of the example tasks are designed to illustrate generic capabilities that are fundamental to any programming system. Here the detailed base functionality of a realistic computer system is not as important. Therefore some tasks list a generic base functionality that is simple enough for a PBD researcher to duplicate in its entirety. The intent is that a PBD system should be able to automate the task relying only on this stated functionality.

Basically, researchers can use the tasks any way they please. For instance, researchers might pick an example that their system does not handle well and redesign their system to handle it better. When designing a new learning algorithm or user interface technique, researchers could scan through the suite and consider how each task might challenge the designs under consideration. If researchers were developing a taxonomy of tasks, they might test that it partitions the Test Suite in a useful way. By varying the base functionality and the task description, researchers can easily generate more examples that can be used to push the generality of their research.

Identifying generic PBD capabilities

One way we hope the Test Suite will be used is to compare how well different PBD systems automate tasks from the suite. Researchers could analyze whether their system can do the task, or what skills and how much effort are required of the user. Those features that consistently contribute to success would be identified and collected into a robust list of generic PBD capabilities. The "Wrap in rounded rectangle" task, which appears later in this chapter with three analyses, is a good demonstration of this idea.

In general, researchers doing analyses should try to stick as close as possible to the tasks and base functionalities stated in the Test Suite. One reason is that comparisons between different PBD techniques will be more meaningful if the assumptions of different analyses are the same. Another is that claims of generality will be stronger; if researchers can show that their techniques apply to the various specific situations of the Test Suite without modification, then they are better prepared to argue that their techniques are general enough to apply to future situations without modification. For instance, the analysis "Triggers automates Wrap in rounded rectangle" shows Triggers automating this one task without modification. If Triggers could consistently achieve this for other examples in the suite and examples contributed in the future, then some claim to generality could be made.

Strictly speaking, few PBD systems will be able to perform the task without modification because many of the realistic base functionalities include applications that are closed systems. This makes it impossible to achieve the data and operator access necessary for most PBD techniques. The same claims to generality can still be made, however, by automating the task on a substitute application using a subset of the functionality of the application stated in the realistic base functionality. The analysis "MatchTool 2 automates Wrap in rounded rectangle" takes this approach. Such demonstrations should strongly motivate application developers to allow the integration of the PBD techniques into future applications.

While there is particular value in sticking to the given base functionalities and tasks, there is also value in analyzing variations. For instance, "Chimera's macro by example facility automates Wrap in rounded rectangle" assumes a slightly different base functionality -- a drawing editor with alignment lines and snap-dragging. On one hand, this can make comparing Chimera's macro by example facility and other PBD systems more difficult. Nevertheless, simply demonstrating the usefulness of alignment lines and snap-dragging to PBD is an important result. Interesting lessons can be learned from studying how PBD systems deal with such variations.

Standardized reporting will help researchers compare their analysis of a task from the Test Suite with analyses by other researchers. The following are recommendations on how researchers should report the results of automating a task with their PBD system. The sample analyses in this chapter adhere to these recommendations.

First the contributions of each of the following should be stated:

Researchers should also discuss how their present implementations add credibility to the analysis. Because most PBD techniques require close integration of the PBD system and the applications, most analyses will be hypothetical to some extent. The relationship between the implementation and their analysis could be one of the following:

Acknowledgments

This chapter was a collaborative effort of the many researchers who contributed example tasks and analyses. Special thanks go to David Kurlander whose tactful challenges led to important improvements in this chapter.

What's in the Test Suite

This first version of the Test Suite contains 32 example tasks and 7 analyses of systems automating them. They appear in alphabetical order, as listed here. We invite readers to contribute more tasks and analyses to our growing repertoire.

Example Tasks

Automation Analyses:

Bold text

Contributed by Richard Potter
See Chapter 17, Triggers

Realistic base functionality

MacWrite II running on a Macintosh
The functionality most likely to be of use is:

Task description

A document from a text formatting language was imported into MacWrite II. Part of the syntax of the text formatting language was to enclose text that should be printed in bold face between the special markers "B<" and ">". Thus, given the sentence

This B<word> should be bold.

the formatter should produce the printed output

This word should be bold.

The task is to scan through an entire MacWrite II document and make the text between special markers bold and then remove the special markers. Assume that no ">"s appear within special markers.

Interesting aspects

This was an actual task that Ben Shneiderman did when writing the second edition of "Designing the User Interface". The first edition was in the text formatting language and the second edition was written with MacWrite II.

Box volume problem

Contributed by William Finzer and Nicholas Jackiw
See Chapter 13, Geometer's Sketchpad

Generic base functionality


Task description

What size corner should be removed from a square piece of paper to maximize the volume of a box made by joining the flaps? The first figure shows the piece of paper with corners removed. In the second we see the paper folded into a box.

This problem might be encountered in a high school algebra or geometry course. In one mode of attack on this problem, the student tries various sized corners, computing the volume of the box for each. The task to automate is the repetitive process of trying various sized corners.


Sample data

The table shows data a student gathered for a piece of paper 1.5 inches wide. 
Size (inches)   Volume (cu.    
                    in.)       
0.1            0.169           
0.2            0.242           
0.3            0.243           
0.4            0.196           
0.22           0.247192        
0.23           0.248768        
0.24           0.249696        
0.25           0.25            
0.26           0.249704

Interesting aspects

Solving math and science problems frequently necessitates repetitive calculation. The student might enter a formula into a programmable calculator or spreadsheet, which constitute mathematical models of the problem. For programming by demonstration, the task presents the challenge of working from one instantiation of the problem situation - one set of corner sizes - and inferring from that the general relationship between the dimensions and the volume.

Changing the format of a bibliography

Contributed by Brad A. Myers
See Chapter 14, Tourmaline;
see also Chapter 11, Turvy

Realistic base functionality

Microsoft Word 4 or 5 running on Macintosh or PC Windows
WriteNow 3 running on Macintosh or NeXT
The functionality most likely to be of use is:

Task description

Given a bibliography of a paper formatted in the CACM style, change it to be in the style used for this book.

Sample data

See Chapter 14.

Interesting aspects

This task can be performed using editing macros in a generic editor, as in Turvy, or using knowledge about the structure of bibliographies, as in Tourmaline. This task would make an interesting test case for comparing the two approaches. You would expect the knowledge-based approach to be much more robust and easy to use.

Complete graph

Contributed by Marc Yvon and Philippe Piernot
See Chapter 18, SPII/AIDE; see also analysis in this Appendix

Generic base functionality

Task description

The task is to connect each selected object to all other selected objects, thus forming a complete graph.

Sample data

See Chapter 18 for an example.

Interesting aspects

This is a simple, intuitive task that can test a PBD system's ability to form nested loops and keep track of sets of items.

AIDE
automates
Complete graph

Analyzed by Philippe Piernot
Task by Marc Yvon and Philippe Piernot
See Chapter 18, SPII/AIDE; the task is described in this Appendix The following shows how AIDE can automate the "Complete graph" task. In this analysis, the SPII graphics editor provides the stated base functionality. SPII was developed using the AIDE toolkit and is therefore able to communicate with AIDE. AIDE needs no modification to accomplish this task.

Demonstration scenario

The detailed scenario is described in Chapter 18.

Contributions by the scenario components

* Base functionality of the application (SPII) * Base functionality of the PBD system * Contribution of the user * Communication between the PBD system and the application * User interface features of the PBD system required for instructing the task

Lessons learned

In SPII, teaching a task is intuitive but the user has to properly order the objects in the selection list; this may be too heavy of a constraint. Showing the trace and enabling the user to undo macros has proven to be very useful.

Compute number of disks

Contributed by Richard Potter

Generic base functionality

Task description

Assume there is an ordered list of files on a hard disk and that all the files are to be copied in the same order to floppy disks. Fit as many files as possible onto each disk. A single file is not to be split across two disks.

The task is to compute the number of floppy disks required to hold a given set of files. Assume that each floppy has the same storage capacity.

Sample data

Size of Floppy: 30 blocks
List of file sizes (in blocks): 6 18 6 21 9 3 18 12 24
The answer should be: [6 18 6] [21 9] [3 18] [12] [24] = 5 disks.

Interesting aspects

This task is interesting because it is a pain to program in most any language owing to numerous opportunities for off-by-one errors. Still, doing it by hand is trivial -- circle the groups of files to be put on each disk and then count the circles. Tricky algorithmically, but intuitively easy to demonstrate on data, this task hints that PBD may be useful in mainstream programming.

When I first encountered this task, my reaction was to try to find a simple solution using the APL programming language, but none of APL's powerful array operators made it any easier. It would be interesting to identify if there are any generic operators that make automating this task easier.

Variation on the task

An interesting, realistic variation is to pick a real computer system and compute the number of floppies required to hold the files in an actual directory when copied in alphabetical order. Then the challenge of having the PBD system access the file sizes from the operating system enters into the problem. Moreover, some files may be too big, necessitating error messages.

Copy matching icons into last folder

Contributed by Dan Halbert
See Chapter 5, SmallStar

Generic base functionality

Task description

Copy all the icons on the desktop whose names contain some substring into the last folder inside a desktop folder.

Sample data

Several icons on the desktop have "Canyon" in their names. Copy them into the last folder (named Recent) inside the folder named Travels.

Interesting aspects

This is a simple-minded task that illustrates several aspects of SmallStar including:

Create a new kind of menu

Contributed by Brad A. Myers
See Chapter 6, Peridot

Generic base functionality


Desired result

Task description

The task is to display a new kind of menu, where each item has its own shadow as shown in the figure. An example list of strings for the menu will be given, but the menu code generated by the system should work for any list of strings. Note that this requires that the system determine or be told how the graphics depend on the parameters (e.g., that the rectangle surrounding the menu depends on the widest item and the height of all the items).

Sample data

Bach, Beethoven, Mozart, Ravel, Strauss

A sample result appears in the figure.

Interesting aspects

Unlike many of the other examples, the task here is for the computer to do something that the user would not normally do by hand. But displaying menus is still a ubiquitous task for an interactive computing system. Given only a user interface software library, a user interface developer could easily implement standard widgets. What makes this task interesting is that it shows how programming by demonstration gives the user interface developer the expressiveness to create a widget with a totally new look and feel. However, this task is challenging for PBD because there must be some way for the system to determine how the graphics depend on the parameters (e.g., that the rectangle surrounding the menu depends on the widest item and the total height of all the items).

Directions arithmetic

Contributed by Ted Kaehler

Generic base functionality

Task description

Build up arithmetic operations on integers that stand for the four directions. Supposing that 1 = up, 2 = right, 3 = down, 4 = left, how can we add directions? Specifically, how can we compute a new direction from the current one plus a given number of 90deg. turns to the right (positive) or left (negative)? We need two functions, one that adds turns to directions, and one that converts direction numbers to the two-dimensional vectors used by a graphics package.

First, define a kind of addition that adds an initial direction and some number of 90deg. clockwise turns, to yield a new direction.

dir + turns = newDir

1 + 2 = 3 up + 180deg. = down

3 + 3 = 2 down + 270deg. = right

4 + 4 = 4 left + 360deg. = left This is the familiar but dreaded function
newDir = ((dir + turns - 1) mod 4) + 1 Second, define a function Vec that computes direction vector components from a direction index. For instance, Vec(3) = (0, 1), where 0 means no change in x and an increase of 1 unit in y (for systems in which y increases downwards).

Interesting aspects

Direction arithmetic comes up in every program that has motion or adjacency in the four directions: chess, board games, video games, street map problems, spreadsheet cells, tables, cellular automata, etc.

Question for researchers

Can your system discover the correct modulo equation without intermediate results supplied by the user?

Draw an arch

Contributed by Henry Lieberman
See Chapter 16, Mondrian

Realistic base functionality

MacDraw II running on a Macintosh
The functionality most likely to be of use is:


Input bounding box rectangle for the "Draw an arch" task

Task description

Start with a rectangle on the screen, to use as a bounding box. Draw an arch, made out of three rectangles, inscribed in the bounding box. The system should learn a procedure capable of inscribing an arch in any box. The width of the columns and the height of the cross piece will be proportional to the corresponding dimensions of the bounding box, and the two columns will have identical dimensions. The bounding box is deleted.


Output of the "Draw an arch" task

Interesting aspects

This is typical of a programming by demonstration task in a graphical editing domain. The user would like to package up sequences of graphical construction commands that depend on the characteristics of some graphical object used as an argument to the demonstration. There are many interesting design choices about which graphical relations to notice (height, width, color, relative versus absolute coordinates, or relations between objects such as alignment, and so on). Design choices about how general to make the resulting procedure will affect what it does when applied to another example. What does it mean, if anything, to make an arch out of a circle rather than a square?

Variation on the task

An interesting variant of the above task is to define a procedure that takes three rectangles and assembles them to form an arch.

Faceplate removal

Contributed by Henry Lieberman

Generic base functionality


Task description

The task is to remove a given faceplate in an assembly with the added condition that all corner fasteners are opened before non-corner fasteners.


A faceplate with all fasteners closed and a faceplate with all fasteners open

Interesting aspects

This task is a very simple example of a repair procedure. A system that could acquire a procedure like this from demonstrations of simulated repairs could serve as a knowledge acquisition tool for on-line repair manuals. The demonstration could be on just one faceplate, but the PBD system could generalize it to any faceplate. Such capabilities could also be used to train robots to do repairs on actual equipment.

An important question is whether a PBD system can use background knowledge to construct generalizations from the demonstration that are meaningful in the domain. Challenges arise when the amount of knowledge required grows large or when the PBD system may have trouble deciding its relevance. Other issues involve how to acquire descriptions of a device and possible actions in a new domain, and how the efficiency of the PBD process compares to other alternatives for specifying these procedures.

Learning about function makes PBD more widely useful. Interaction with simulated devices is a step toward programming by demonstration for virtual reality systems.

Fractal snowflake

Contributed by David Kurlander
See Chapter 12, Chimera; see also analysis in this Appendix

Generic base functionality

Task description

A triadic Koch curve, often simply referred to as a fractal snowflake, appears in the figure. The task is to draw this shape - or rather, a good approximation to it, since the precise shape contains an infinite number of segments.
Fractal snowflake


Interesting aspects

The snowflake contains the same basic pattern at many orientations and scales. It is difficult to draw good approximations of this shape by hand, because of the vast numbers of lines that need to be positioned and sized precisely. This task has been cited as an example of why programming interfaces are useful in graphical editors [Asente 87].

MatchTool 2
automates
Fractal snowflake

Analyzed by David Kurlander
Task by David Kurlander
See Chapter 12, Chimera; see also [Kurlander 88a];
the task is described in this Appendix This section explains how to use MatchTool 2's graphical search and replace facility to automate the "Fractal snowflake" task. An unmodified version of MatchTool 2 can automate this task exactly as specified, in the Chimera drawing editor. This analysis assumes that Chimera supplies only the base functionality listed in the task description.

Demonstration scenario

We first draw an equilateral triangle in the editor scene, using either constraints, alignment lines, or a polygon tool to make the drawing precise. Next, we replace each segment, regardless of scale and orientation, with a certain trajectory of four segments. To make a better approximation to a fractal snowflake, we repeatedly invoke search and replace. We can set up the replace pattern to leave other graphical properties unchanged (for instance, line color and line width), or we can have these properties copied from the example replacement objects.

Contributions by the scenario components

The scenario components are as described in "MatchTool 2 automates Wrap in rounded rectangle", except that no constraint facilities are used.

Lessons learned

This example shows graphical search and replace being used to create a shape formed by a graphical grammar. Using Chimera and graphical search and replace, the user can perform this task without programming. Graphical search and replace has other applications outside of typical graphical editing. It can produce complex illustrations from simple ones, using graphical templates. Also, graphical search is useful as an iteration mechanism for macros by example in graphical editors.

List-manipulation procedures

Contributed by Henry Lieberman
See Chapter 2, Tinker; see also Chapter 1, Pygmalion

Additional reference:
Henry Lieberman and Carl Hewitt, "A Session with Tinker: Interleaving Program Testing With Program Design." Proceedings of the First Lisp Conference, Stanford University, August 1980.

Generic base functionality

The functionality equivalent to the following operations as in Common Lisp:

Task description

Define simple list-manipulation primitives, such as appear in introductory texts on Lisp, for instance:

Sample data

Input Output

(reverse `(A B C D)) (D C B A)

(append `(A B) `(C D)) (A B C D)

Interesting aspects

These simple procedures test how well a PBD system handles classic programming constructs, including conditionals and recursion. Handling these effectively would be particularly important for PBD systems used to replace conventional textual programming environments for simple symbolic or numerical programming tasks or for other domains that are not in themselves particularly graphical or demonstrational.

List of mail subjects

Contributed by Allen Cypher
See Chapter 9, Eager

Realistic base functionality

HyperCard 2.1 running on a Macintosh
The functionality most likely to be of use is:

Task description

Given a stack of mail messages, where the first line of each one begins with "Subject: " followed by the subject of the message, create a numbered list of the subjects from each card in the stack.

Sample data

Suppose you have a stack of cards, such that:

Line 1 of field "Message" of Card 1 is "Subject: Trial info"

Card 2 is "Subject: Some more good ideas"

Card 3 is "Subject: a necessary evil..."

Card 4 is "Subject: Fitness Center re-opens"

Card 5 is "Subject: Where were you?"

The result should be a list in the field "Mail subjects":

1. Trial info

2. Some more good ideas

3. a necessary evil...

4. Fitness Center re-opens

5. Where were you?

Interesting aspects

Existing macro tools (like Tempo II and QuicKeys2) cannot automate this task effectively because

Mail merge

Contributed by Dan Halbert

Realistic base functionality

Microsoft Word 4.0 running on a Macintosh
The functionality most likely to be of use is:

Task description

Copy addresses from a text file opened in one window into a form letter in another window, without using any existing "mail-merge" capability. Print the letter after each copy. Addresses are separated by blank lines and may be variable numbers of lines long (e.g. 3 lines or 4 lines). There's a specific place in the form letter that will receive the address, but there is no particular requirement on how this space is indicated to the PBD system.

Sample data

A three line address and a four line address to be inserted into a form letter:

Daniel C. Halbert
Digital Equipment Corp.
Cambridge MA

Allen Cypher
Advanced Technology Group
Apple Computer, Inc.
Cupertino CA

Interesting aspects

This is a simple realistic task that I've wanted to do several times. Even though many word processors have built-in facilities for "mail-merge", in simple instances, it may be easier for the user to automate on-the-fly with a PBD system than to learn how to use the more sophisticated built-in facility. This could serve as a design goal for making PBD systems easy to use. This task also has opportunities for real-world complications. For example, suppose printing fails? How do you report the error clearly?

Natural language processing

Contributed by Henry Lieberman
See Chapter 2, Tinker

Additional reference:
Henry Lieberman, An Example-Oriented Environment for Beginning Programmers, in Artificial Intelligence and Education, Bob Lawler and Masoud Yazdani, eds., Wiley, 1987.

Generic base Functionality

Functionality equivalent to standard Common Lisp


Simple context-free grammar

Task Description

Implement a parsing procedure that takes a list of words as input, and produces a parse tree as output according to the simple context-free grammar in the margin.

Sample Data

Input: (a spider saw the man with the telescope)

Output: (S (NP (ART a) (NOUN spider)) (VP (VERB saw) (NP (NP (ART the) (NOUN man)) (PP (P with) (NP (ART the) (NOUN telescope))))))

Interesting Aspects

The linguistic literature is full of example sentences that can be used to motivate and test development of parsing strategies. Example-oriented techniques should be natural for the development of language understanding programs, and the needs of language understanding programs are applicable to many other areas of artificial intelligence. For "extra credit", use examples to demonstrate the intermediate stages in parsing. If you have a parsing system that produces multiple parsings for ambiguous sentences such as the one above, use multiple examples to demonstrate how ambiguous parses are handled. Animate a tree display of the parsing process.

Odd/even pages

Contributed by Richard Potter

Realistic base functionality

Microsoft Word 4.0 running on a Macintosh with System 7
and LaserWriter driver 7.0
The functionality most likely to be of use is:

Task description

When I print out a long document, I like to print it on both sides of the paper. The way to do this is to print the even pages in reverse order, then reinsert the paper into the LaserWriter and print the odd pages in forward order. With Microsoft Word 4.0, you can only print consecutive ranges of pages, so this involves issuing a separate print command for each page.

The task is to print all even pages in reverse order and at a later time print all the odd pages in forward order.

Interesting aspects

This is a simple realistic task that should be easy to specify by demonstration. The task is useful enough that it may eventually be included as a feature, but the fact that the feature has not existed for a long time helps justify why systems should enable users to create these features for themselves using PBD. (Interestingly, during the time this book was being written, this feature finally appeared in Microsoft Word.)

Outline heading numbers

Contributed by David Maulsby

Realistic base functionality

Microsoft Word 4 or 5 running on Macintosh or PC Windows
WriteNow 3 running on Macintosh or NeXT
Only the most basic editing functions (insert and delete text) are needed.

Task description


Sample data for the "Outline heading numbers" task. The desired result appears below the line.

Suppose you have an outline with Roman numeral headings and lettered subheadings indented one tab. An example appears in the sidebar. Convert the numbering to Arabic and align the heading text, as shown. Note that heading numbers end with a period, subheadings do not. If the title runs over more than one line, subsequent lines are indented to the same tab marker as the first line.

Interesting aspects

Some editors (like EMACS) can do regular expression searches, but do not automate the insertion of numbers in series (unless you write a system extension). Others (like MS Word) can do the renumbering -- once you have converted all lines to appropriate styles, which may conflict with other uses of those styles in the document.

One issue this task illustrates for PBD is that there can be several reasonable ways for the user to demonstrate the task. For example, the user could (1) process all the main headings first, then all the subheadings, and finally indent the wraparound lines, or (2) process the text line by line; at each line, do the appropriate edit. PBD systems should take this into account.

There are many variations on this task:

Phone number formats

Contributed by Ted Kaehler

Realistic base functionality

HyperCard 2.1 running on a Macintosh
The functionality most likely to be of use is:

Task description

Take a phone number that could be in any of following different formats and translate it into the form "basic 7-digit number", area code (see sample data). When the area code is 415, the form "basic 7-digit number" should be used. This standard form then can be used by automatic dialing software.

Sample data

555-1212 -> 555-1212

314 555-1212 -> 555-1212, 314

(314) 555-1212 -> 555-1212, 314

[314] 555-1212 -> 555-1212, 314

314- 555-1212 -> 555-1212, 314

1,314- 555-1212 -> 555-1212, 314

415- 555-1212 -> 555-1212 -- strip off my local area code

Interesting aspects

The numerous special cases make this task really several tasks blended into the same program. It should be possible to build a complex program up from simple cases. There needs to be a way to say, "Good so far, now expand the program to work on this case as well."

Perhaps the only way to build this by example is in small steps.

Pong

Contributed by Henry Lieberman
See Chapter 2, Tinker

Additional reference:
Henry Lieberman, Video Games by Example, CHI `85 video tape [SigGraph Video Review #12]


Before state. This illustration shows a situation where the player Henry has missed hitting the ball as it is about to hit the wall.


After state. Since Henry missed the ball, Richard scored a point. The ball returns to the center of the board to begin a new round.

Generic base functionality

Task Description

The task is to instantiate the artifacts and maintain dynamics of the computer game Pong. The game consists of a rectangular board, one ball, and two paddles, one on the left side of the screen, the other on the right. A scoreboard lists each player's name and score, initially 0. A paddle is a short vertical line, whose Y position is controlled by joystick input from a human player. The ball has a constant velocity, and a heading that can be changed by collision with objects. When the ball collides with a paddle or with a wall of the board, its direction changes, using the rule that the angle of incidence equals the angle of reflection. Each player tries to hit the ball, that is, to move the position of the paddle so that it collides with the ball when the ball approaches. If the ball does not hit the paddle, and the ball hits the wall behind the player's paddle, the score of the opposing player is incremented. This situation is illustrated in the figures. After each point is scored, the ball returns to the center of the screen, and is given a random heading. If one player attains a maximum score, the game ends.

Interesting Aspects

The video game example is important because it takes programming by demonstration into a dynamic domain. Issues here include how to program dynamics in terms of state-transition functions, and how to deal with interactive user input. Showing configurations of playing objects in "before and after" states of the game would be an intuitive way to program the game by demonstration. Also, video games are fun!

Precision floor plan

Contributed by Alan Turransky
See Chapter 24, Voice Input; see also Chapter 16, Mondrian


The empty box

Realistic base functionality

Canvas 3 or MacDraw II running on Macintosh
Adobe Illustrator 3 running on PC or Macintosh
The functionality most likely to be of use is:


Initial sketch of desk block

Task description

During the initial stage of laying out a floor plan, architects typically block out the design to get a feel for the overall layout before they include detail. Rectangles are used to represent specific objects or groups of objects (called blocks) such as tables, chairs and doorways in a layout. The task is to precisely create and place a block representing a table into the top-left corner of the box. It should be one-third the width of the box and equal to it in height, as shown.


Align to top-left

Interesting aspects

No matter what the content of the drawing is about, easy to use and compatible input tools for creating precision drawings make it possible for users to quickly test out ideas. In this task the user needs to input constraints while drawing. It is desirable that this be seamlessly integrated into the act of sketching an object. Inference from an example and concurrent voice input are two alternative techniques for accomplishing this.


Width one-third, height same

Rearranging a page layout

Contributed by Henry Lieberman
See Chapter 16, Mondrian

Realistic base functionality

PageMaker 4.0 running on a Macintosh
The functionality most likely to be of use is:

Task description

Given a layout of a front page of a newspaper, teach a command to swap two non-overlapping, equal-size blocks of items. For example, the sample data shows the right half of the page (minus the header) swapped with the left half of the page (minus the header).

Sample data


Input to the "Rearranging a page layout" task Output from the "Rearranging a page layout" task

Interesting aspects

This example is representative of a graphical edit that is frequently repeated, so that a user would benefit from automation. It exercises editing commands on existing objects, rather than the construction of new objects exercised by the "Draw an arch" task.

Reconnecting components of a balance

Contributed by David Kurlander
See Chapter 12, Chimera; see also analysis in this Appendix

Generic base functionality

Task description

A picture of a balance appears in the figure. To change the angle of the balance's arm, we need to rotate the two segments of the arm about the fulcrum, and reattach the trays. We would like the system to perform these tasks for us automatically whenever we adjust some component of the illustration.
The balance


Interesting aspects

Constraints can be used to relate the geometric properties of scene objects to one another, but it can be tricky to figure out all of the constraints that should be specified. In this example, the user might not think of fixing the length of the trays' strings, or fixing the orientation of the trays to be horizontal.

Also, solutions to this problem should work no matter which part of the drawing we initially adjust. This makes using a procedural approach difficult, since the steps would differ according to which object is moved first.

Chimera
automates
Reconnecting components of a balance

Analyzed by David Kurlander
Task by David Kurlander
See Chapter 12, Chimera; see also [Kurlander 91];
the task is described in this Appendix This section analyzes how Chimera can automate the "Reconnecting components of a balance" task by inferring the necessary constraints from multiple valid configurations of a scene. Chimera can automate this task without modification.

Demonstration scenario

We start with the picture of a balance, and we press a button to take a snapshot of this initial configuration. Next we rotate the segments forming the arm of the balance, reconnect the trays to the arm, and take a second snapshot. Now we can turn on constraints, and Chimera enforces these relationships. Illustrations of this example, and a more detailed description, appear in Chapter 12.

Contributions by the scenario components

Various components of the scenario contribute to automating this task: * Base functionality of the application (Chimera) * Base functionality of the PBD system * Contribution of the user * Communication between the PBD system and the application * User interface features of the PBD system required for instructing the task

Lessons learned

Constraints facilitate graphical editing by maintaining desired invariant relationships among objects as illustrations are drawn and modified. Since it is often difficult to figure out which constraints to add to an illustration, research has addressed methods of inferring constraints. Peridot's object-object inferencing rules (Chapter 6), and Chimera's constraint-based search and replace (Chapter 12) infer constraints from a static picture. However, it helps to know how objects move relative to one another, so Chimera can also infer constraints by examining multiple valid configurations of a scene, and determining which relationships do not change.

The constraints that Chimera inferred here from multiple examples would have been impossible to infer from a static picture of the balance, without domain knowledge. There would be no way of knowing that the arm of the balance rotates about a fulcrum, yet the trays remain upright. There is no indication from a static picture that the base remains fixed in space, yet the trays move. However, when the user is allowed to demonstrate multiple valid configurations, the system can more readily infer such relationships.

Chimera performed this task without difficulty, however sometimes there are disadvantages with this approach. Chimera can infer incidental constraints: invariant relationships in the snapshots that the user did not intend. Also, the system usually finds redundant constraints, which are constraint relationships expressed multiple times, but in different ways.

Several other examples of inferring constraints from multiple examples are given in [Kurlander 91]. These examples include inferring constraints in a desk lamp, and how interface components change when a window is resized.

Reformatting an
address list

Contributed by Ian Witten
See Chapter 8, TELS; see also Chapter 14, Tourmaline; and Chapter 11, Turvy

Realistic base functionality

Microsoft Word 4 or 5 running on Macintosh or PC Windows
WriteNow 3 running on Macintosh or NeXT
The functionality most likely to be of use is:

Task description

Start with a file of addresses in which each entry, consisting of name, address and phone number (in that order), appears on a single line. Reformat it so that each entry is separated by a blank line and the name, street address, city, and postcode are on separate lines, and the phone number does not appear.

Sample data

See Chapter 8 for an example.

Interesting aspects

The task is simple but realistic. It requires learning different address formats (e.g. with and without apartment numbers) and the format of various numerical items (e.g. telephone numbers, ZIP codes).

It is a genuinely difficult programming challenge using conventional techniques (although utilities exist, they do a poor job) and it provides a clear example of the need for incremental, example-driven, software development and of the difficulty (even futility) of specifying a particular format fully in advance.

Repetitive calculator operations

Contributed by Ian Witten
See Chapter 3, Predictive Calculator

Generic base functionality

A non-programmable hand calculator, or simulation thereof
The functionality most likely to be of use is:

Task description

To calculate y = xe1-x for a dozen or so values of x.

The values of x are specified by the user. The calculator must pause after each calculation with the result displayed.

Interesting aspects

This is a very simple, but realistic, task. It can be inferred from examples with no knowledge of the semantics of calculations or the calculator.

Variation on the task

Strings may be repeated within different higher-level functions, representing a sub-formula or repeated use of a variable value. For instance, "1 + a2 - 2a cos" is repeated twice in the following formula.

20 + 10 log BBC[(1 + a2 - 2a cos F(180x,4000)) - 10 log [1 + a2 - 2a cos 45]

Replacing terminals with workstations

Contributed by David Kurlander
See Chapter 12, Chimera; see also analysis in this Appendix

Generic base functionality

Task description

The user has drawn an illustration of a computer network, in which identical pictures of computer terminals appear many times. The pictures of computer terminals in the initial document were not structured in any special way. The graphical objects (polylines and arcs) forming the terminals were not grouped together, and no special terminal object was defined as part of the drawing process. Change all of the pictures of terminals in the network diagram to workstations.

Interesting aspects

This task consists of two steps: recognition and substitution. Graphical objects composing the terminal illustrations must first be identified, and then replaced with other objects forming pictures of workstations. The task is complicated by the user not having grouped or classified terminal objects when drawing the illustration.

MatchTool 2
automates
Replace terminals with workstations

Analyzed by David Kurlander
Task by David Kurlander
See Chapter 12, Chimera; see also [Kurlander 88a];
the task is described in this Appendix This section explains how to use MatchTool 2's graphical search and replace facility to automate the "Replace terminals with workstations" task. An unmodified version of MatchTool 2 can automate this task exactly as specified, in the Chimera drawing editor. This analysis assumes that Chimera supplies only the base functionality listed in the task description.

Demonstration scenario

This scenario is described and illustrated in Chapter 12. We copy graphical objects forming one of the computer terminals from the illustration into the search pane of MatchTool 2. We draw a workstation in the replace pane, or copy it from some other source. In the first column of checkboxes, we specify which attributes of the objects in the search pane will be important for our search, and in the second column we specify which attributes of the object in the replace pane will be applied to the match when a replacement is made. In this example, we choose to match and replace on shape, as well as many other graphical attributes, such as fill color and line width.

Contributions by the scenario components

The scenario components are as described in "MatchTool 2 automates Wrap in rounded rectangle", except that no constraint facilities are used.

Lessons learned

Graphical search and replace finds all objects in an illustration matching a set of graphical properties, and alters some of these properties. Even though the graphical primitives forming the terminal are not representationally grouped together as a terminal object, graphical search and replace can still automate this task because it can match on shape.

Graphical search and replace is a demonstrational interface, because the user works with example objects to specify operations to be performed on more general classes. Here, the user generalizes the example explicitly by indicating which properties of these objects are significant.

Re-routing

Contributed by Alan Turransky
See Chapter 24, Voice Input; see also the "Xs" task in this Appendix

Realistic base functionality

Canvas 3 or MacDraw II running on Macintosh
Adobe Illustrator 3 running on PC or Macintosh
The functionality most likely to be of use is:

Task description

You have a drawing editor with a number of rectangles. A few of the rectangles are connected by lines to form a chain. The task is to re-route the lines such that the centermost rectangle is substituted with a selected rectangle not already in the chain. If a chain has an even number of rectangles, the program should tell the user that there is no one centermost rectangle, and let the user choose one of the two nearest the center.

Interesting aspects

This task involves counting and following a connected structure. An analog of this task is to re-route a flight through a different connecting city on an interactive map. A system could learn it from one example of three cities if the user can give an instruction to disambiguate "center" from "second" in the chain.

Shorten lines

Contributed by Richard Potter
See Chapter 17, Triggers

Realistic base functionality

MacDraw II running on a Macintosh
The functionality most likely to be of use is:

Task description


A drawing and the same drawing with lines shortened to the edge of the first intersecting object

There are many situations in graphic drawing where the endpoint of a line should extend precisely to the edge of some other graphic object. Sometimes this relationship is created by making the lines too long and then shortening them to the proper length.

The task is to take a given endpoint of a line and shorten it until it meets the edge of the first intersection object (see the figure).

Interesting aspects

This is a realistic task that I find myself doing manually when I rough out tables. By the time the contents of the table are set, I have inevitably moved most of the lines out of alignment.

The task is interesting because it is easy for the user to demonstrate but it is hard to compute. Thus it would be difficult for a PBD system to generate an algorithm that can compute the intersection point between the line and the edge of the object.

Another reason this task is interesting is that the intersection point is clearly represented in pixel form on the computer display. For certain special cases of this task, it is easy to compute the intersection point from pattern matching on this pixel representation.

Sort a bar chart

Contributed by David Maulsby
see Chapter 7, Metamouse

Realistic base functionality

Canvas 3 or MacDraw II running on Macintosh
Adobe Illustrator 3 running on PC or Macintosh
Only basic objects (boxes) and functions (select and drag) are needed.

Task description


Sample data for sorting a bar chart. The desired result appears at the bottom.

Suppose you used a graphing program to plot a bar chart from spreadsheet data, and then resized and decorated the chart using your favorite drawing program. After pasting it into your word processor file, you realize that the text would read more clearly if the bars were sorted by increasing height from left to right. Rather than start over from scratch with the spreadsheet, you sort the boxes using PBD and the drawing program.

The task is based on the experience described above, but altered a bit to make it more nasty for inference-based PBD systems. Before and after snapshots are shown in the figure. Sort the boxes so that height increases from left to right. If two boxes are the same height, the wider one goes to the right. If they are also the same width, the order is arbitrary. The space between each box is constant. Text labels remain at the same relative location underneath each box.

Note that in the before snapshot, the first four boxes in order of increasing height (P, B, D, I) are aligned at the top and evenly spaced with respect to their centers; this should mislead inference-based systems to predict G (even spacing) or E and S (same height) out of order.

Interesting aspects

This task illustrates another dimension of the data description problem often overlooked by PBD systems -- the ordering of a set of objects by the values of some property. And in this case there are nested orderings (height and width). Sorting also raises the issue of whether a PBD user specifies an algorithm or a set of constraints. It would be quite unnatural and difficult for most users to teach a bubblesort or quicksort; more likely the user would select a few in order, or show how two rectangles would be swapped, then let the PBD system do the real task of sorting. The other issue raised by this task is how a user might specify a numerical ordering without leaving a purely graphical mode of work.

Surnames

Contributed by David Maulsby
See Chapter 11, Turvy; see also Chapter 14, Tourmaline

Realistic base functionality

Microsoft Word 4 or 5 running on Macintosh or PC Windows
WriteNow 3 running on Macintosh or NeXT
Only the most basic editing functions (insert and delete text) are needed.

Task description

This is a slightly altered version of Task 3 in the Turvy study.


Sample data for the "Surnames" task. The desired result appears at right.

Put authors' surnames before their given names and initials, as shown in the figure. Make sure that a comma separates each surname from its initials, and each author from the next. A single period followed by a blank space should separate the last author from the title (avoid double periods after initials).

Telling where the list of authors ends may be difficult; there may be any number of authors. Note that titles are always either italic or quoted. The figure shows various cases to handle. There may be one given name or several, with zero or more initials. Watch out for the annotations "(ed)" or "(eds)" after the last surname in some examples. Retain baronial prefixes ("de", "le", "l'", "van", "van der"). In a more difficult variant of the task, some entries ("Michalski" in the example), may be partly done already.

Interesting aspects

People often create text with a fairly regular structure that editors do not capture. Bibliographies are notorious for being regular yet so full of special cases that reformatting them is exasperating, even with special-purpose tools such as EndNote and BibTex. Automating this task is further complicated by conflicts between special cases and errors in the data. It requires nested iteration to process multiple authors in multiple bibliographic entries. It is also challenging because the PBD system must maintain some point of reference within text that remains reliable while the parts of the names are moved around.

This task suggests the virtue of learning special cases versus general patterns. Coming up with a general grammar for names is a formidable task (consider having to include names as varied as "Tupou IV", "Cher", "H.D." and "Fr. Jean-Pierre de la Roche, S.J., PhD.") Generalized patterns would be apt to classify "H.D." and "S.J." as initials requiring a surname to complete them, but one is a person's full (professional) name and the other designates a person's vocation. "IV" is not a variant form of initials. "Cher" is a complete name. It would be all-to-easy for a learning system to overgeneralize, and all-too-difficult for a user to anticipate the special cases.

Word paste

Contributed by Richard Potter

Realistic base functionality

Microsoft Word 4.0 running on a Macintosh
The functionality most likely to be of use is:

Task description

When a user selects a group of words and pastes it into another sentence, sometimes the number of blanks before and after the newly pasted text must be adjusted to ensure that one and only one blank separates the words. Punctuation introduces special cases because no spaces should come between a word and a following comma or period.

The task is to appropriately adjust the number of spaces after the user pastes in a string of text. Assume that commas and periods are the only special cases. The pasted text may contain extra spaces, but it will not contain partial words.

Sample data

Paste "good" at the | in "We need more| movies."
Paste "good " at the | in "We need more| movies."
Paste "more good " at the | in "We need |movies."
Paste " more good " at the | in "We need |movies."
Paste "good movies " at the | in "We need more|."
Paste " good movies" at the | in "We need more|." All these should produce "We need more good movies."

Interesting aspects

This task is interesting because it is a simple real world task to which most anybody can relate, yet it is complicated enough to include some special cases which could be challenging to demonstrate. Word 4.0 does not have a tag or mark feature, so the PBD system will have to keep track of the beginning and end of the newly pasted text somehow.

Wrap in rounded rectangle

Contributed by Richard Potter
See Chapter 17, Triggers; see also analyses in this Appendix

Realistic base functionality

MacDraw II running on a Macintosh
The functionality most likely to be of use is:

Task description

MacDraw II can create text fields and it can create rounded rectangles, but there is no way to have the size and location of the rounded rectangle automatically adjust to the size and location of the text field.

The task is to create a rounded rectangle that is slightly larger than a given text field and place it so that it wraps around the text field.

Interesting aspects

A fellow lab member discovered that this task was tedious to do manually when he was mocking up a user interface design with MacDraw II and decided to represent buttons with text fields wrapped in rounded rectangles.

Two text fields and the same text fields wrapped with rounded rectangles




MatchTool 2
automates
Wrap in rounded rectangle

Analyzed by David Kurlander
Task by Richard Potter
See Chapters 12, Chimera; and 15, Macros by example; and [Kurlander 92b];
the task is described in this Appendix


This section explains how to use MatchTool 2's constraint-based search and replace to automate the "Wrap in rounded rectangle" task. MatchTool 2 can automate this task without modification in the Chimera drawing editor. The only base functionality provided by the Chimera editor that does not exist in MacDraw II is geometric constraints. Since the constraint functionality could be freely moved to MatchTool 2, this analysis shows that had MacDraw II included a PBD component with the power of both MatchTool 2 and the constraint solving functionality, then it could allow users to automate the "Wrap in rounded rectangle" task.


The contents of the search and replace panes are shown. To make the figure less cluttered, only the constraints relating to the top right vertices of the text and rectangle are displayed. Symmetric constraints are present on the other vertices, yielding a total of 12 constraints.

Demonstration scenario

To perform this task, we make use of MatchTool 2, described briefly in Chapter 12 and more extensively in [Kurlander 92b]. In the search pane we type any text, and check only the object class property in the search column so that any text string in the scene is considered a valid match.

Next we copy the text in the search pane to the replace pane, and place a rounded rectangle around it. When the rectangle is first added to the scene, it appears above the text, so we execute a command to change the overlap order.

Next we add geometric constraints to both objects in the replace pane to precisely adjust the geometry of the rectangle in relation to the text. Since the text should not move when we add subsequent constraints to it, we place fixed location constraints on all of the text's vertices (often called handles). Next we add distance constraints between each vertex of the rectangle and the corresponding vertex of the text. Finally we add angle constraints between each vertex of the rectangle and any two vertices of the text. For example, we create a 135 degree angle constraint between the top right vertex of the rectangle, the top right vertex of the text, and the text's top left vertex.

This collection of constraints completely specifies the desired margin and orientation between the text and the new rectangle. We check constraints in the replace column, so that MatchTool 2 alters only constrained geometry during the replacement and adds new objects constrained to existing objects (in this case the rectangle is added).

The search and replace pattern is now defined to match on any text and place a rounded rectangle around it. Several other different, but semantically equivalent, sets of constraints could have been chosen instead. The user can invoke this newly defined replacement everywhere in the scene, or at a particular location. This rule can be saved in an archive for future use.

Contributions by the scenario components

Various components of the scenario contribute to automating this task: * Base functionality of the application (Chimera) * Base functionality of the PBD system * Contribution of the user * Communication between the PBD system and the application * User interface features of the PBD system required for instructing the task

Lessons learned

In contrast to using Triggers for this task (see below), the disadvantage of the constraint-based approach is that the application and PBD extension mechanism must be more tightly coupled. However, it has the advantages of being less sensitive to screen state (background color, location of the text on the screen, nearby objects), and it can place a rounded rectangle around text at any rotation, oriented in the same direction as the text.

The hardest challenge for the user is figuring out which set of constraints to instantiate in the replacement pattern. It is not necessary to get it right the first time, since it is easy to experiment with the pattern, and make incremental modifications to it.

As discussed in Chapter 12, the same general mechanism can be used to find all nearly connected lines, and connect them. Other examples described elsewhere [Kurlander 92b] show how this technique can make lines nearly tangent to circles precisely tangent, round the corners of polylines by splicing in arcs, and beautify a roughly drawn illustration of a house.

Chimera's macro by example facility
automates
Wrap in rounded rectangle

Analyzed by David Kurlander
Task by Richard Potter
See Chapters 12, Chimera; and 15, Macros by example;
the task is described in this Appendix Using the Chimera editor, the "Wrap in rounded rectangle" task can be automated in a second way. This alternate approach relies on Chimera's macro by example facility and the editable graphical history representation of command sequences. The macro by example facility can perform this task without modification in Chimera, but assumes a slightly different base functionality than stated in the task description because alignment lines and snap-dragging functionality are assumed. Therefore this analysis shows that had MacDraw II included a PBD component with the power of Chimera's macro by example, and if MacDraw II included alignment lines and snap-dragging, then it could allow users to automate the "Wrap in rounded rectangle" task.

Demonstration scenario

Initially we perform the task on an example. We create a text string, and select it to generate distance alignment lines. These lines are a kind of drafting tool used for precise positioning in cooperation with the snap-dragging user interface technique [Bier 86]. Since we would like a margin of 1/4 inch between the box and the text, we create a set of alignment lines 1/4 inch from the edges of the selected text. The drawing cursor is attracted to these lines, and more so to the intersections of multiple lines. We draw a rounded box using these lines, thus giving it the desired margin. Next we select the box, and execute a command to move it below the text.

Sometime after completing this initial example, we realize that the steps performed earlier can be encapsulated into a useful macro. We open up a graphical history window, then find and select the relevant panels. Next we execute a command to generate a Macro Builder window containing these panels.

We would like the string to be an argument to the macro, so we make the panels editable, select the string, give it a name, and execute a command that makes it an argument. A panel appears at the beginning of the Macro Builder window showing the argument declaration. In a similar fashion, we select the widget from a history panel that specified the 1/4 inch margin, and make this margin value an argument too.

Argument specification is one step in generalizing the macro to work in other contexts, but further generalization may be needed. We can invoke our macro now and the system will automatically assume a most likely generalization for each command. Alternatively, we can select panels, view their default generalizations, and change them. In this example, Chimera infers the intended generalization without user intervention.

Now we can name the macro, save it, and invoke it elsewhere. When the macro is invoked, a window is generated to prompt the user for arguments. We choose a new string and margin width as arguments, and the macro executes correctly on the new example.


The graphical representation of a macro to wrap a rounded rectangle around a text field. The first two panels show the argument declarations and the remaining panels illustrate the macro actions.

Contributions by the scenario components

Various components of the scenario contribute to automating this task: * Base functionality of the application (Chimera) * Base functionality of the PBD system * Contribution of the user * Communication between the PBD system and the application * User interface features of the PBD system required for instructing the task

Lessons learned

The advantages and disadvantages of this approach are similar to those of using constraint-based search and replace: details of screen state will not interfere with the macro, however the extension mechanism and application must be more tightly integrated.

While demonstrating the task, we used alignment lines. This made the positioning of the rounded rectangle relative to the text apparent to the PBD system when it came time to generalize the operations. In this example we used alignment lines as a type of tool. The use of tools to make intent explicit is investigated in Chapter 7. In working with Chimera, we have found alignment lines to be particularly useful tools in macro specification for four reasons:

1. They are useful in editing; people need not learn their use just for PBD.

2. There are several different kinds of alignment lines, parameterized by slope, angle, or distance. These parameters can be made parameters to the macro. For example, above we used 1/4 inch distance lines to create a margin of this size, and later we specified that this measurement be one of the macro's arguments.

3. Most alignment lines are infinite in length, so the tool used during demonstration will handle new contexts encountered during execution.

4. Since alignment lines are measurement tools, they can be trivially generated to precise dimensions. Typically, greater effort is required to construct scene objects with precise measurements.

However, alignment lines cannot be created with any shape, so they cannot always replace scene objects as tools.

Triggers
automates
Wrap in rounded rectangle

Analyzed by Richard Potter
Task by Richard Potter
See Chapter 17, Triggers;
the task is described in this Appendix The following is an analysis of how Triggers can automate the "Wrap in rounded rectangle" task. Triggers needs no modifications to automate this task, because it can communicate with MacDraw by reading pixels of the Macintosh display screen and by sending user interface events to MacDraw II. The analysis shows that an application does not need to be modified to serve as the basis for new automation, but shows several places where Triggers' user interface could be improved.

Demonstration scenario

A detailed scenario of how a user would program this task using Triggers is given in Chapter 17. The basic strategy is to take advantage of the fact that MacDraw II will outline a selected text field with a gray box. By searching for pieces of this box, Triggers' pixel pattern matching can determine the locations of the upper-left and lower-right corners of the gray box. A correctly sized rounded rectangle can be created with mouse actions defined relative to these locations.

Contributions by the scenario components

Various components of the scenario contribute to automating this task: * Base functionality of the application * Base functionality of the PBD system * Contribution of the user * Communication between the PBD system and the application * User interface features of the PBD system required for instructing the task

Lessons learned

The most noteworthy point about this analysis is that Triggers can automate this task without modifications to either Triggers or MacDraw II.

The worst part about specifying the algorithm is placing the rectangles that define the pixel patterns and the search areas. In total, eight rectangles have to be placed, so any way to reduce the tedium of placing these rectangles would be a significant improvement to Triggers.

The device-level algorithm is robust enough to be useful, but is limited in a number of ways. The entire text field must appear on the screen at one time, which is not a big deal. The background around the text field must be white, which is more of a problem. Moreover, gray lines around the text field can be confused with the text field's bounding box. This analysis does not explore the important question of what class of users can create device-level algorithms of this complexity.

Xs

Contributed by David Maulsby
See also the "Re-routing" task in this Appendix

Realistic base functionality



Sample of data for the "Xs" task. The desired result is at the bottom.

Canvas 3 or MacDraw II running on Macintosh
Adobe Illustrator 3 running on PC or Macintosh
Only basic objects (boxes) and functions (select and set fill pattern) are needed.

Task description

You have made a wallpaper design consisting of X shapes, as shown in the figure. You want to blacken all the squares on the diagonal from the upper left to lower right, except for the one at the center, which is changed to white.

Interesting aspects

This task seems rather hard to program without PBD, unless you can make some strong assumptions about where Xs are located and how squares are spaced within them. The task may well prove difficult for PBD users to program. The PBD system must learn to identify the end of a chain of boxes, otherwise it may recolor a gray box in the opposite diagonal of another X. It must single out the center square, which is a member of two diagonal chains, or a box with four other boxes nearest to it; either of these patterns is non-trivial to derive from conventional representations of graphical objects.

back to ... Table of Contents Watch What I Do