There aren’t very many qualitatively different types of computer interfaces in use in the world today. But with the release of Mathematica 9 I think we have the first truly practical example of a new kind—the computed predictive interface.

If one’s dealing with a system that has a small fixed set of possible actions or inputs, one can typically build an interface out of elements like menus or forms. But if one has a more open-ended system, one typically has to define some kind of language. Usually this will be basically textual (as it is for the most part for Mathematica); sometimes it may be visual (as for Wolfram SystemModeler).

The challenge is then to make the language broad and powerful, while keeping it as easy as possible for humans to write and understand. And as a committed computer language designer for the past 30+ years, I have devoted an immense amount of effort to this.

But with Wolfram|Alpha I had a different idea. Don’t try to define the best possible artificial computer language, that humans then have to learn. Instead, use natural language, just like humans do among themselves, and then have the computer do its best to understand this. At first, it was not at all clear that such an approach was going to work. But one of the big things we’ve learned from Wolfram|Alpha is with enough effort (and enough built-in knowledge), it can. And indeed two years ago in Mathematica 8 we used what we’d done with Wolfram|Alpha to add to Mathematica the capability of taking free-form natural language input, and automatically generating from it precise Mathematica language code.

But let’s say one’s just got some output from Mathematica. What should one do next? One may know the appropriate Mathematica language input to give. Or at least one may be able to express what one wants to do in free-form natural language. But in both cases there’s a kind of creative act required: starting from nothing one has figure out what to say.

So can we make this easier? The answer, I think, is yes. And that’s what we’ve now done with the Predictive Interface in Mathematica 9.

The concept of the Predictive Interface is to take what you’ve done so far, and from it predict a few possibilities for what you’re likely to want to do next.

In Mathematica 9, the way this works is that when you have an output in focus, a Suggestions Bar appears below it, with a list of buttons for possible actions to take next. What buttons appear is determined by a computation that is run in real time when you request the Suggestions Bar.

There are two kinds of inputs to this computation. The first is what actions are common given the structure of the output, and the earlier history of your session. And the second is what actions would lead to useful results.

Over time, the Predictive Interface will be able to learn from the actions people take in it. But to get started, we’ve used several large sources. The first is the collection of carefully tuned heuristic algorithms that Wolfram|Alpha uses to determine what pods to output given a particular input. The second is the billions of actual queries that we’ve seen in the Wolfram|Alpha query stream. The third is published Mathematica code—for example the huge number of examples in the Wolfram Documentation Center and the Wolfram Demonstrations Project. And the fourth is our very large internal sample of Mathematica code from the source code of both Mathematica itself and Wolfram|Alpha.

From the query stream and heuristic algorithms in Wolfram|Alpha we learn many a priori probabilities for actions to be taken on different kinds of objects, as well as a certain amount about sequences of actions. From samples of Mathematica code we learn what kinds of actions occur together—for example what the probabilities are for different functions to appear applied to different kinds of arguments.

In a first approximation, what happens in the Predictive Interface is that possible outputs are categorized into hundreds of different general types, each represented by a symbolic expression that encodes certain properties and attributes. Then, informed by our various sources, large numbers of probabilistic rules are set up to determine conceivable actions that might be suggested for particular combinations of types. Which actions actually make sense to suggest will then depend on which ones lead to useful results. So typically what the Predictive Interface does is to try out candidate actions—or tests based on them—and then use heuristics to assess the utility of the results they would give.

The Predictive Interface will often internally find quite a large number of possible actions to suggest. These then have to be ranked so that the best choices can be presented first. And the way this is done is through a fairly elaborate system of scores, that combine specific heuristic algorithms with probabilistic information and assessments of the utility of results.

But after all this sophisticated computation, what the user ultimately sees is just a simple list of buttons for possible actions to take.

Let’s look at an example:

The Predictive Interface is suggesting a few things to do with this integer. And they seem pretty sensible. But actually they’re even better than we might expect. Because the Predictive Interface is using information it gets by actually doing the computations it’s suggesting. Let’s try another integer:

Again the suggestions seem pretty sensible. But they’re different. And the reason is that the Predictive Interface knows that this particular integer is a prime. So it can tell for example that a primality test will be particularly worth doing on it. Sometimes it can be quite uncanny how prescient the Predictive Interface manages to be. But what I’ve found is that when one gets used to this, it’s surprisingly useful not only in its primary purpose of guiding one through what one can do, but also in giving implicit hints about features of what one’s seeing.

It’s interesting to compare the concept of the Predictive Interface with the way Wolfram|Alpha generates reports. In Wolfram|Alpha, given a particular input—like an integer, for example—Wolfram|Alpha will generate a sequence of “pods” which it displays down the page, giving results of computations that its heuristic algorithms determine are interesting. In the Predictive Interface there’s also selection of computations going on, but now not for the purpose of actually displaying the results of these computations. Instead, after all sorts of internal work, all that’s actually displayed is a sequence of buttons, presented for the human user to pick what to do next.

Of course, one can combine these approaches, and when the Predictive Interface determines that it’s going to be useful, it “lights up” the Wolfram|Alpha logo in the Suggestions Bar. If you then press this, you’ll get the whole Wolfram|Alpha result—from which you can pick an individual pod to generate a specific new Mathematica input.

The Predictive Interface makes suggestions that lead to many different kinds of actions. But what we’ve found is that it’s best to present every suggestion in a more or less uniform way—by having one or more plain English words on what is effectively a button. Sometimes pressing the button may just apply some individual Mathematica function, like Solve (for equation solving), or FactorInteger (for integer factorization). And in this case, the next input makes it immediately obvious what was done:

Sometimes the Predictive Interface may end up composing pieces of code on the fly, which again can be applied with individual buttons:

Often these pieces of code are simple enough that it is not distracting to display them completely. But sometimes it’s better to “hide” them by default, with an “opener” provided if one wants to actually look at the code:

And sometimes the code is actually an invocation of Wolfram|Alpha:

The Predictive Interface ranks suggestions, and immediately displays the top few. If you press “more…”, you’ll get a panel, which will typically show many more suggestions, now arranged in categories:

Sometimes there’s just one form of a particular suggestion. But quite often there are alternatives or options. When there are a fairly small number of easy-to-understand choices, the Predictive Interface just lists them by name in a pull-down:

Or, if what’s going on is more complicated, it shows previews of what the result would be for different choices:

In many cases, there isn’t just a list of possible choices; instead one may have to “fill in a form” to say what one wants:

In general, the Predictive Interface can present an essentially arbitrary user interface. Internally, it’s just generating a symbolic Mathematica expression, which can then make use of anything in the Mathematica interface language—for example to give a custom-created “wizard-like” panel:

In simple cases, the Predictive Interface just gives a simple row of suggestions buttons. But it’s fairly common for there to be very different kinds of suggestions depending on what a particular output is supposed to mean. And in such cases we’ve found that instead of mixing suggestions based on different meanings, it’s much better just to pick a default meaning, then offer other meanings as explicit alternatives—analogous to the way Wolfram|Alpha’s “assuming” mechanism works:

When one uses the Predictive Interface, it’s pretty common to end up with a whole string of inputs and outputs. The way the Predictive Interface is set up, each new input contains the previous output. Here’s a sequence, with the explicit Predictive Interface Suggestions Bars for each line added in:

If you want to repeat the sequence of computations here, you typically want to “roll them up” into a single line—which is what the spiral icon in the Suggestions Bar does:

Right now, the Predictive Interface concentrates on making suggestions for single outputs—though it often makes use of context and previous history. In the future, we’re planning to do a lot more on multi-input suggestions, and various forms of refactoring, as well as on full Mathematica programs.

When we started building the Predictive Interface it was not at all clear it was going to end up working out. Previous examples of “suggestions” interfaces had generally not been well received (think for example Microsoft’s “Clippy” intelligent paperclip—which I have to say I always found charming, if not especially useful). But I suspected the problem was not the general idea of providing contextual suggestions, but the way they were being generated and presented.

And the key idea that’s led to our Predictive Interface for Mathematica 9 is to put real computation into figuring out what to suggest. There’s never going to be a complete, precise, algorithm to determine what a person is going to want to do next (though people are often much more predictable than one might expect). And instead, what one needs to do is to have a whole collection of heuristic algorithms that come as close as possible to being able to make a computer “do what I mean”.

I have to say that in past years I was always skeptical about heuristics. Because I thought people would find them very frustrating. When one has a precise language and system like Mathematica, the essence of good design is to make everything completely consistent, so people can readily predict what the system will do in a particular case. But heuristics go in the opposite direction, trying to cover common cases well, but not worrying at all about overall consistency.

But here’s a key point I’ve learned from creating Wolfram|Alpha: if heuristics are done well, with serious computation and knowledge behind them, they actually do work, and people like them very much. Wolfram|Alpha is absolutely full of heuristics: for understanding free-form linguistic input, for deciding what output to generate, etc. And—as it is so often with computer systems—so long as everything “just works”, people never think about the heuristics, never try to deconstruct them, and never notice or get confused by the lack of ultimate consistency.

The Predictive Interface is technically rather different from Wolfram|Alpha. But the notion of having a whole web of heuristics based on serious computation and knowledge is the same. In practice with the Predictive Interface it’s also important that it presents itself with the appropriate level of emphasis: it’s there, and easy to get at if one wants it, but understated enough that it doesn’t visually get in the way if one doesn’t need it.

And I have to say that in my own use of Mathematica it seems to work well. If I know what to do after I get a particular output, I just type the next input without getting distracted by the Predictive Interface. But as soon as I pause even for a moment, I tend to glance at the Predictive Interface. And often it jogs my thinking, and gives me exactly what I want to do next.

I’ve now had a chance to watch a few Mathematica beginners use the Predictive Interface. It seems to work really nicely. It both gives them the satisfying experience of making progress more quickly than ever before, and it gently exposes them to a wider range of capabilities in Mathematica that they might not otherwise discover for a long time.

Mathematica is a big enough system that I don’t think anyone (even myself) can immediately remember everything it does. So that means that particularly if one is using a somewhat unfamiliar part of the system, the Predictive Interface is highly useful. And even in areas of the system that I know well, it’s just faster to press a Predictive Interface button than to type an input.

The Predictive Interface consists of a kind of infinite web of suggestions. And it’s rather fun to try starting with something simple, and seeing just how far one can go simply by following Predictive Interface suggestions. It’s remarkable how quickly one can end up doing some pretty sophisticated things.

One of the big things I’ve always tried to do in Mathematica (and in Wolfram|Alpha and so on) is to automate as much as possible: to make it so that whatever the computer can automatically handle it does automatically handle. In the past, we’ve done a lot on automating the selection of algorithms, automating the way output and visualizations are presented, or interfaces are built, and, in Wolfram|Alpha, automating the way input is interpreted. With the Predictive Interface, we’re attacking yet another “automation frontier”: automating how one chooses what to do next.

From the point of view of interface design, I am finding the Predictive Interface extremely interesting. Previous interfaces—like menus or forms or computer languages or free-form linguistics—make certain kinds of things easy. The Predictive Interface makes a new set of things easy. And as I work on future design for Mathematica—as well as other products of ours—I can already see my thinking changing as a result of the Predictive Interface.

In language design one typically wants to have a minimal number of names and concepts for people to remember. In free-form linguistic input one wants to support whatever people will immediately think of—and it’s barely worth covering things that people will never “think to ask”. But with the Predictive Interface one has a new mechanism. Once people are going in a particular general direction, one can present to them suggestions that let them discover things that they’d never think were there, or were even possible.

This is particularly important for a system like Mathematica that is deep and broad. It’s too easy for people to spend years using just a small part of the system, and never get the benefit of its wider capabilities. But now the Predictive Interface constantly leads people to other parts of the system that they can immediately use and become familiar with.

The Predictive Interface in Mathematica 9 is just the beginning. In time, it will become possible to do still much richer and more sophisticated predictions. Making use not just of information from the current session, but all sorts of history, data and analytics about the user. The direction is in a sense maximal automation. To have the user define a goal, but then have the computer figure out as much as possible about how to achieve that goal. Sometimes the user will be able to specify the goal using natural language or computer language. But often they will not have formulated it completely enough to do so. And instead they’ll just be able to state a general direction to go. At which point the Predictive Interface can take over, making suggestions and letting the user guide the computer in the direction they want to go.

Today the Predictive Interface is available in Mathematica 9. We already have other products in the works that make use of it. And in the future I expect to see sophisticated computed predictive interfaces show up all over the place—defining in a sense a new paradigm for interacting with computers.