back to ... Table of Contents Watch What I Do


Just-in-Time Programming

Richard Potter


Many of the other chapters have presented advancements in programming by demonstration (PBD) by presenting PBD systems and their innovations. In other words, these chapters have presented solutions. This chapter takes another tack by discussing PBD in the context of a problem. The problem is to create a new type of programming system that overcomes the obstacles users encounter when they attempt to use present-day programming systems for just-in-time programming. This chapter defines just-in-time programming and identifies five of these obstacles: inaccessible data and operators, the effort of entering the algorithm, limited computational generality, effort of invoking the algorithm, and risk. Just-in-time programming motivates PBD research because PBD can potentially overcome several of these obstacles.

Just-in-time programming is the implementing of algorithms during task-time, the time when the user is actually trying to accomplish the task. It can be characterized by a situation with the following components:

In short, the goal of just-in-time programming is to allow users to profit from their task-time algorithmic insights by programming. Instead of automating with software that was carefully designed and implemented much earlier, the user recognizes an algorithm and then creates the software to take advantage of it just before it is needed, hence implementing it just in time.

It is worth emphasizing that the user's task could be from any domain (e.g. graphic drawing, scientific visualization, word processing, etc.) and that the algorithm originates with the user. Obviously, a user with more programming experience will be able to envision a more complex algorithm than a novice user. How the user comes up with the algorithm is not a concern. Also, no particular approach to solving the problem appears in the problem statement. Any programming system could conceivably be used for just-in-time programming, including C, PASCAL, keyboard macros, scripting languages, or PBD. PBD will probably be an important part of the more successful just-in-time programming systems, but the problem statement leaves open the possibility for other solutions.

Just-in-time programming research shares many of the motivations of other PBD research. Chief among these is that users often do repetitive or algorithmic subtasks that the computer could be doing. We call these subtasks potential computer subtasks and call these situations opportunities for new beneficial automation. Because automating can increase productivity and user satisfaction and at the same time reduce errors, one would expect the user to delegate potential computer subtasks to the computer. That users often do not take advantage of these opportunities motivates researching ways to improve the computer. Just-in-time programming research and PBD research assert that easier-to-use programming tools will allow users to take better advantage of opportunities for new beneficial automation.

Just-in-time programming research, however, is focused on making programming easier for a specific cross section of situations. These situations are primarily defined by the user programming during task-time. In other words, the user attempts to write a program for a task that is already in progress. Figure 1 summarizes the relationship between task progress and the user's expenditure of effort. The expenditure of effort for just-in-time programming is shown separately from the other task related effort. The difficulty of just-in-time programming results from the spreading of the user's mental resources between two activities [Cypher 86]. Another difficulty is that the time spent programming contributes directly to total time between the start and completion of the task.

Figure 1. Two scenarios are shown of a user presented with an opportunity suitable for just-in-time programming. The intermixing of programming effort with other task related effort is shown for the second scenario where the user decides to apply just-in-time programming.

A Subtask Suitable for Just-in-Time Programming

Opportunities for new beneficial automation that are suitable for just-in-time programming arise in many situations in interactive computing. There will always be repetitive subtasks that slip through the prepackaged functionality of applications because they result from the interactions of users with the complexities of the real world. Task-time is often the only possible time to implement the algorithms that can automate these subtasks. The benefits of improved just-in-time programming systems would be to allow users to better automate these repetitive subtasks that arise from their unique circumstances.

A classic example is when a document in one format has to be transformed to another format because of circumstances beyond the user's control. A small example of this happened to me when Symantec Corporation updated their Think C class library to version 1.1. Before version 1.1, rectangles were defined with 16-bit coordinates and in version 1.1 rectangles were defined with 32-bit coordinates. When I first compiled my software document with the new class library, type mismatch errors occurred where my software expected a variable representing a 16-bit rectangle. Many of these were simple assignment statements. The new class library included a utility function for converting 32-bit rectangles to 16-bit rectangles, so a typical fix involved changing a line of the form *inset=frame; to the form longToQDRect(&frame,inset);. Various other types of errors were found and fixed as well. The second time an assignment of a 32-bit rectangle to a 16-bit rectangle caused an error, I recalled that there were many such assignments throughout my program and concluded I would, in time, be transforming many lines from assignment statements into function calls. Each would differ only in the names of the variables and whether each variable was a pointer or not (i.e. preceded by a "*"). For the rest of this chapter, transforming one of these lines will be called the line transformation subtask.

So to break this situation down into the components of just-in-time programming:

myself modifying a software document to work with an updated class library changing certain lines of source code from the form

{*}var1 = {*}var2;

to the form


(i.e. the line transformation subtask)

insert leading white space

insert "longToQDRect("

if second variable name is not preceded by "*", insert "&"

insert second variable name

insert ","

if first variable name is not preceded by "*", insert "&"

insert first variable name

insert ")"

insert rest of line (the ";" and comments, if any)

delete original line.

I considered a couple of alternatives for automating the subtask, but decided to transform the lines manually. The rest of the chapter will explain why.

Five Obstacles

It is instructive to analyze present-day programming systems to understand why a new type of programming system is desired for just-in-time programming. This section presents a user-centered analysis by identifying five obstacles that users frequently encounter when they attempt just-in-time programming using present-day programming systems. The obstacles are: The five obstacles help summarize the reasons why present-day programming systems are frequently ineffective, show multiple areas where just-in-time programming research can be directed, and provide a check list for analyzing future programming systems. It only takes one obstacle to prevent a user from profiting from an algorithmic insight. Because these obstacles are common, researchers must address all five if their systems are to be widely effective.

Inaccessible data and operators

The first obstacle is inaccessible data and operators. Users encounter this obstacle when their programming system is unable to observe or manipulate the parts of the computer critical for automating the subtask. The data could be information in their document (e.g. values in spreadsheet cells), or the state of a user interface widget (e.g. the position of a scrollbar). The operators could change a document (e.g. delete a word in a word processor), or change some other aspect of the computer (e.g. eject a floppy disk or enlarge a window). The root cause of this obstacle is that subtasks suitable for just-in-time programming typically involve processing data that users are manipulating within application programs like word processors, spreadsheet applications, and drawing editors. These applications have historically been closed systems and most programming systems have been unable to reach inside them to gain access to data and operators. Data and operator access is so fundamental that this obstacle alone will often thwart attempts to automate a tedious subtask by programming.

For the line transformation subtask, data access meant obtaining the erroneous line's location and contents so that the algorithm could modify the part of the file that contained the line. Because of the way the compiler reported finding an erroneous line, this information clearly existed in the program editor. The compiler would automatically highlight each newly detected erroneous line in the program editor application, where I would normally edit the line manually. Because the program editor was a closed application, most of the programming systems on the computer were unable to access the required information. This was especially true of many versions of traditional programming languages like C and PASCAL, which typically can only access the file system and the user interface devices.

PBD helps eliminate the data and operator access obstacle both directly and indirectly. PBD indirectly eliminates the data and operator access obstacle simply by motivating research; PBD depends on data and operator access being possible and thus motivates the development of interapplication communication protocols like Apple Events. PBD takes a more direct role by greatly facilitating the use of low-level constructs in programs, which can sometimes allow data and operator access into otherwise closed applications. For example, the keyboard macro solution that appears shortly achieves operator access by simulating low-level user actions at the keyboard. Triggers (Chapter 17) shows how to extend this approach to data access using the pixels on the computer screen. PBD makes these solutions practical by providing the contextual clues and feedback that enable users to relate low-level data and operators to corresponding higher-level meanings.

Effort of entering the algorithm

The second obstacle is the effort of entering the algorithm. This obstacle entails all the mental and physical effort users must expend to transfer their algorithmic understanding of the subtask into the programming system. Since users will inevitably make some errors, the effort users expend detecting and correcting these errors is included too. However, the effort of forming algorithmic understandings of subtasks is not included because just-in-time programming assumes users already have this understanding.

For the line transformation subtask, this obstacle never became an issue because the inaccessible data and operators obstacle alone prevented automating the subtask. But had data and operators been accessible, the effort of entering the algorithm would be a serious obstacle with many programming systems. For example, assume the program editor had a scripting language as extensive as the HyperTalk scripting language of HyperCard. Figure 2 shows a HyperTalk script that will automate the line transformation subtask, assuming that the line has been isolated in a variable. To make use of this HyperTalk script, the user would have to create it based on their algorithmic understanding of the subtask and physically enter it into the scripting system's editor. Just the physical effort of typing these 1185 characters (692 characters if comments and leading blanks are not included) is likely to undermine the benefits of automating. So while textual programming languages like HyperTalk are perfectly good programming languages for many purposes, they are often too verbose to be effective just-in-time programming languages.

Reducing this obstacle is PBD's forte. PBD helps reduce the effort of entering the algorithm by allowing users to enter algorithms using the same interface they would normally use to perform subtasks manually. This helps reduce both the physical effort and the mental effort because the user is often well practiced at using this interface. Since users would use the same interface to perform subtasks manually, the artifacts are already in short term memory and programming with them is likely to be less distracting than with a textual programming language. The effort of entering the algorithm is also reduced because user interfaces are usually optimized to the task.

Figure 2. A HyperTalk function that automates the line transformation subtask assuming the line has been isolated in a variable

For a simple example of how PBD can dramatically reduce the effort of entering an algorithm, consider the following partial solution to the line transformation subtask. If the user first places the cursor to the left of the first variable in the line to be transformed, and if neither variable is a pointer, then the QuicKeys macro shown in Figure 3 will transform the line as required. The macro also assumes that exactly three characters (" = ") separate the two variable names. Only the 33 keypresses shown in Figure 3 are required to implement the macro. The visual feedback of the editor also helps reduce the mental effort by showing intermediate results as the user demonstrates the low-level keystrokes that accomplish the subtask.

Limited computational generality

The third obstacle is limited computational generality. Users encounter this obstacle when their algorithms require control constructs that are not supplied by their programming systems. It is important for a just-in-time programming system to have full Turing-complete computational generality because there is no way to predict which of the vast array of possible algorithms users might envision being useful. Full computational generality is, in some sense, easy to achieve because only conditional and looping control structures are required. However, many programming systems trade computational generality for ease of use. Users who attempt just-in-time programming frequently face a dilemma: those programming systems that are potentially the most suitable for just-in-time programming do not offer the computational generality necessary to automate their subtasks.
Figure 3. This QuicKeys macro can partially automate the line transformation subtask. The number of keystrokes required to enter the macro and the visual state of the editor are shown.

Limited computational generality is the reason why the virtues of the keyboard macro were demonstrated by only partially automating the line transformation subtask. The subtask requires conditional logic to decide whether each variable is a pointer or not, as specified in the algorithm. Most keyboard macros only record straight-line algorithms and are not able to fully automate this subtask.

Unfortunately, computational generality is not one of PBD's strengths. Dan Halbert recognized this when implementing SmallStar and concluded that control structures were better created by editing a static representation of the program than by demonstration (Chapter 5). Others have used inference to generalize straight-line demonstrations into procedures that have control structures. Allen Cypher's Eager (Chapter 9) and Brad Myers' Peridot (Chapter 6) used domain knowledge to infer procedures with control structures solely from straight-line demonstrations. The computational generalities of these systems, however, are limited by domain knowledge.

In order for a PBD system to be widely effective for just-in-time programming, it will have to be integrated with other techniques to give full computational generality. Interesting directions include giving separate examples for each path of the algorithm as in Henry Lieberman's Tinker (Chapter 2), or a combination of multiple demonstrations, inferencing, and special instructions from the user as in David Maulsby's Metamouse (Chapter 7).

Effort of invoking the algorithm

The fourth obstacle is the effort of invoking the algorithm. This obstacle entails all effort users must exert to take a program they have already implemented and put it into use. This includes judging if the automation is appropriate for the particular situation, remembering how to invoke the algorithm, and exerting the physical effort of actually invoking it. Users must not only implement their algorithm during task-time, but must also give it a user interface that allows them to invoke it easily enough to gain benefit from each invocation of the automation.

The line transformation subtask is typical of many subtasks suitable for just-in-time programming in that the benefit gained from each invocation is small. One popular invocation strategy, especially with keyboard macros, is a specially assigned keypress. One keypress to transform one of the lines is a satisfactory solution, and certainly does not present an obstacle if just the physical effort is considered. The effort sometimes becomes an obstacle when complexities of the task-time situation plus the effort to remember and properly apply the automation strain the user's attention. Keystrokes can become difficult to remember because most simple keystrokes are already claimed by application programs. Often only convoluted keystrokes like control-option-"L" are available. If several opportunities appropriate for just-in-time programming arise, keeping straight which keystroke goes with which automation can be a challenge.

There are situations where even minimizing the physical effort can be particularly crucial towards making automation beneficial. Consider the feature on many word processors that allows a user to select a word simply by double-clicking on it. The word processor automatically does the tedious subtask of extending the selection out to the word boundaries. Identifying these word boundaries manually is a simple subtask, so not much benefit is received each time the feature is used. However, words are selected so commonly that, over time, the feature is very beneficial. Another invocation strategy could easily undermine this benefit. For example, even requiring the user to click on the word and then select the feature from a pull-down menu could require too much effort.

Just-in-time programming systems should allow the user to choose among various invocation strategies. Standard invocations such as menu selections and keypresses should be supported. The ability to create more refined invocations, like double-clicking on an object to apply some automation to it, would be important for making some highly interactive automations worth creating. PBD techniques could possibly be used to demonstrate that the automation should be invoked whenever the user starts to perform the subtask. David Maulsby's Turvy (Chapter 11) and Metamouse (Chapter 7) give hints as to how this might work. See Chapter 21 for further discussion of invocation techniques.


The fifth obstacle is risk. Users encounter many aspects of risk when they attempt just-in-time programming. Implementing the algorithm may take longer than expected. The envisioned algorithms might not handle future instances of the subtask. Given the pressures of the overlying task and the intrinsic complexities of programming, the user may not have time to evaluate the risks. A user who is considering a just-in-time programming effort has the option of continuing to work manually. Given that many things can go wrong, it is not surprising that the user would choose this option. Thus a programming system often fails for just-in-time programming because the user chooses not to use it.

Consider the risks of automating the line transformation subtask. There are many possible scenarios. In the best case the algorithm could have been entered almost effortlessly, and as each occurrence of a line needing the simple transformation was flagged by the compiler, I could have easily invoked the algorithm somehow. To my surprise, perhaps more chances to use the new automation occurred than were anticipated, making the automation pay off more than expected.

But there are many other possible scenarios. The algorithm could have taken a long time to enter, perhaps because some special purpose function had to be looked up in a manual. A mistake in the implementation might have caused the new (not beneficial) automation to destroy part of the source file, perhaps too quickly to be noticed. Limited data access could have turned the simple algorithm into one that was impossible to implement. I was not sure exactly how many more assignments of 32-bit rectangles to 16-bit rectangles were left in my software project, and thus there may have been too few to make the programming effort worthwhile. Unforeseen special cases may have made the envisioned algorithm simply wrong.

Risk was the main reason I chose not to automate the line transformation subtask. The partial solution using keyboard macros was the only solution worth considering because it was the only one that had ready data and operator access. In the past, my attempts to use keyboard macros have often been thwarted by unforeseen special cases, the difficulty of accommodating special cases into an already existing macro, and the uncontrollable speed of macros that makes it difficult to verify that the macro works correctly. In retrospect, a keyboard macro would have been worthwhile and would have prevented a few recompiles caused by typos in my manual transformation of the lines. However, at the time, the apparent risks convinced me to play it safe and transform the lines manually.

Essentially the user's risk is that the manual method might be more effective than implementing the algorithm. Therefore, an approach to reducing risk is to allow the user to pursue both alternatives in parallel. In theory, the risk of attempting to automate the subtask would be eliminated because if unforeseen difficulties make the programming effort ineffective, then the user could fall back on the manual method already underway.

In practice, this approach would probably not eliminate risk, but it could reduce risk greatly. PBD could play a large part in realizing this approach because it allows the user to implement algorithms by demonstrating on their actual task data. In other words, the user can be programming and manually accomplishing the subtask simultaneously. For example, recording the keyboard macro in figure 3 actually transforms one of the lines, so progress towards completing the overall task is hindered minimally.

This technique has its greatest potential when mixed with history-based techniques. For example, Allen Cypher's Eager (Chapter 9) records the user's actions into an event history. When Eager detects the user doing repetitive actions, it indicates this to the user by highlighting what it expects the user to select next. For certain classes of algorithms, the user can implement an algorithm at almost no risk because the user takes no special actions. The decision of whether to invoke the algorithm still involves some risk because the exact behavior of some algorithms is difficult to predict. Therefore additional techniques such as undo and slow motion execution will have to be extended and refined.


Just-in-time programming requires a new type of programming system because users frequently encounter at least one of the five obstacles when using existing programming systems. With a traditional programming language, users are likely to encounter inaccessible data and operators. With scripting languages, users are likely to encounter the effort of entering the algorithm. With macros, users are likely to encounter limited computational generality. Often these systems introduce multiple obstacles to the user. In particular, risk is almost universally a significant obstacle.

The five obstacles outline multiple avenues of research. The effort of entering algorithms and limited computational generality encourage researching ways to balance PBD with techniques that ensure full Turing-complete computational generality. The effort of invoking the algorithm encourages designing flexible invocation schemes that will not interfere with the existing user interfaces of applications. Inaccessible data and operators encourages establishing new communication protocols or finding ways to take advantage of existing protocols. Risk encourages researching PBD techniques that allow the user to flirt with the possibility of automating a subtask without hindering their progress on the overlying task. Researching ways to overcome the obstacles individually gives the freedom to investigate and evaluate these and other more speculative research directions.

When it comes time to produce practical just-in-time programming systems, all of these obstacles must be addressed simultaneously. If compromises need to be made among which of the five obstacles to address, eliminating risk should receive the highest priority. If users cannot accurately assess the risks, the programming system will fail for lack of use. The goal should be to create a programming system where users can know that in the worst case, attempting just-in-time programming will not hinder progress towards completing the task. Then users will be able to confidently use the full extent of the programming system to profit from their algorithmic insights.

PBD promises to play an important part in making effective just-in-time programming a reality. It is the most promising research area for addressing risk and the effort of entering the algorithm. It plays supportive roles with the effort of invoking the algorithm and the inaccessible data and operators. The open question is whether PBD combined appropriately with other techniques can produce a programming system that eliminates all the obstacles.


This research was funded by Apple Computer. I would like to thank Allen Cypher for encouraging a simplified presentation of this work, Ted Kaehler for contributing his HyperCard expertise, and Andy Sears for helpful comments.

back to ... Table of Contents Watch What I Do