"Watch What I Do" Chapter 11


Chapter
11

The Turvy Experience:
Simulating an Instructible
Interface

David Maulsby

Introduction

An instructible computer acquires data descriptions and procedures by being taught rather than programmed. Teaching requires less planning, analysis or understanding of the computer's internal representation, but it raises problems of how to correlate, generalize and disambiguate instructions. Human pupils interact with their teacher to solve these problems; a computer, with its more limited (and rather different) abilities, will need help specific to its "cognitive style". An instructible user interface involves more (and more flexible) interaction than a programmable or purely demonstrational one, and will fail if it confuses or frustrates the user. To explore the possibilities and pitfalls, we had end users help us design and test interaction methods for an instructible agent, called Turvy. Turvy behaves in some ways like a human apprentice, yet has primitive background knowledge and limited language understanding. We wanted to refine and test ideas before implementing them, so we used the Wizard of Oz technique; Turvy was a researcher hidden behind a screen. We (that is, our users) rapidly found simple, effective ways of teaching through demonstration and verbal hints, while Turvy found ways to elicit them.

Figure 1 shows a sample task and dialog between Turvy and one of our users.

Figure 1. Task 3: Make a heading with author's name and date. This task is one of the most difficult, since it involves parsing lists of people's names, which may include initials, baronial prefixes like "van", and special annotations like "(ed.)". Turvy must learn to find the primary author's surname -- the word before the first comma or colon. It may include a lower case word (like "van"). The date is the last two digits before the period at the end of the paragraph. In some cases the final period is missing; Turvy must ask about this exception. In two of the entries, the author's name and initials are reversed ("Michalski R. S."): Turvy therefore picks the last initial. When the user corrects it, Turvy asks why this case is treated differently, and the user must teach it to skip back over single-letter words ending in a period.

c. Kate's transcript

Kate: Take the last name, it will be in some cases before a colon.
[Kate copies "Agre", pastes it to a new paragraph, types brackets and sets the style.]
This is a list of publications. Now we're going to add the last two digits of the date [types "88"].

Do you want another demo?

Turvy: I can try if you want.

Kate: OK.

Turvy: Looking for word before first colon in next paragraph [picks "Andreae"], OK?

Kate: OK.

Turvy: [while doing actions] Copy, go to start of paragraph, OK?

Kate: OK.

Turvy: New paragraph, open bracket, paste, close bracket, change style to "citation", OK?

Kate: OK.

Turvy interprets hints about syntax (before a colon) but not about subject matter (name, publications, date). Turvy matches hints with actions, taking "last two digits" to mean those she typed, not those at the end of the paragraph.
Turvy interprets Kate's question to mean the example is completed. Turvy has formed an initial generalization and offers to edit the second entry ("Andreae" in Figure 1). Each step that involves relocating the cursor is treated as a pattern search. When Turvy finds an instance of the pattern,
it does a macro, describing actions so Kate can verify or correct them.

Turvy: Type in blank space "88", OK?

Kate: No, the last two digits at the end.

Turvy: Show me please.
[Kate picks digits.]

Two digits before period at end of paragraph?

Kate: Yes.

Turvy: May I continue?

Kate: OK.

Now Turvy makes a mistake with the date. When Kate typed "88," Turvy found no instance of it in recent actions, so it assumed the number is a constant. Turvy always requires a demo to go with verbal instructions. After Kate shows the right action, it echoes back a new hypothesis.

[Turvy picks "Smith"]

Kate: No - stop.

Turvy: Show me please.

Kate: I want the last name of the first author.

Turvy: I don't understand - show me please.

Kate: When you have a number of authors, choose the first [picks and copies "Andreae"]. It's before the comma.

Turvy: Word before first comma or colon?

Kate: Yes.

This next example introduces a new case, multiple authors (see the third entry in Figure 1). Turvy's search pattern was "word before first colon in the paragraph," which selects the last author. Note that Turvy rejects the generalization "any punctuation" for two reasons: words before periods ("E.", "H.") were not given as examples; and Kate said "comma."

[Turvy picks "S."]

Kate: No.

Turvy: Show me please.

Kate: I want you to take in this case the very first word [picks "Michalski"],
then continue as before.

Turvy: You skipped back over words ending in period?

Kate: Yes.

Turvy: Continue?

Kate: OK.

A little later on, Turvy comes to an entry in which the surname happens to precede the initials ("Michalski" in Figure 1). Turvy picks the last initial. Kate's hint "first word" does not repair the pattern, so Turvy proposes the pattern "word followed by 0 or more words ending period, followed by comma or colon".

[Turvy picks "Lehn"]

Kate: OK, um, stop [hesitant]. In this case I want you to select the last two words.

Turvy: Show me please.

[Kate picks "van Lehn"]

What's different about this case?

Kate: Hmm [confused].

Turvy: You selected these two words, why?

Kate: Because these two words make up his last name.

Turvy: Is there some feature of that word [highlights "van"] I should look for?
That word exactly?

Kate: [hesitant] In this case, yes.

Finally, Kate teaches Turvy to parse names with baronial prefixes ("van Lehn" in Figure 1). Turvy recognizes "case, select, last two words" in Kate's hint. "Case" suggests a new special case, or the case of the word. The latter explains "van Lehn," but the Wizard plays dumb. Turvy asks for some feature that completes the pattern: (0 or 1 of ??) followed by word before (0 or more words ending in period) before first comma or colon.

In the dialog presented above, Kate's goal is to get Turvy to do the task (until a new case requires further teaching), and Turvy's goal is to get more specific information from Kate about what patterns to look for. Typically, Kate does one example, then Turvy does the next. When Turvy errs, Kate demonstrates the correction and maybe gives a hint; Turvy echoes back its interpretation. If Kate's hint is ambiguous, Turvy proposes a guess to elicit further explanation. This sample session also illustrates the role of the Wizard, who has to interpret Kate's actions and hints according to pre-defined rules, while using his discretion to make Turvy a little extra stupid if that helps gather more experimental data.


Turvy

Uses and Users

Application domain: Generic

Intended users: End users

User Interaction

How does the user create, execute and modify programs?
The user performs the task by hand at least once, giving a verbal explanation if desired. Then Turvy tries to perform the task, asking the user for assistance and explanation when needed.
Turvy supports incremental development -- the user can modify the program as new cases arise.

Feedback about capabilities and inferences: Turvy announces when it first matches an action; describes search patterns; describes generalizations.

Inference

Inferencing:
Turvy creates programs that are sets of Find&Do pairs. For text-editing, it can find patterns that are chunks of styled text bounded by delimiters, and it can find a context based on nearby words.

Types of examples: Multiple examples, Negative examples.

Program constructs: Variables, loops, conditionals.

Knowledge

Types and sources of information:
Knowledge about forming procedures
Application-specific knowledge about objects (e.g. text can have an "Italics" property), forming search and replace patterns, and the types of arguments required by application events.

Implementation

Machine, language, size, date: Wizard of Oz experiment, 1990.

back to ... Table of Contents Watch What I Do