Note: There have been additional updates to Mathematica. Read about the updates in Version 11.1, Version 11.2 and Version 11.3

I’m thrilled today to announce the release of a major new version of Mathematica and the Wolfram Language: Version 11, available immediately for both desktop and cloud. Hundreds of us have been energetically working on building this for the past two years—and in fact I’ve personally put several thousand hours into it. I’m very excited about what’s in it; it’s a major step forward, with a lot of both breadth and depth—and with remarkably central relevance to many of today’s most prominent technology areas.

It’s been more than 28 years since Version 1 came out—and nearly 30 years since I started its development. And all that time I’ve been continuing to pursue a bold vision—and to build a taller and taller stack of technology. With most software, after a few years and a few versions, not a lot of important new stuff ever gets added. But with Mathematica and the Wolfram Language it’s been a completely different story: for three decades we’ve been taking major steps forward at every version, progressively conquering vast numbers of new areas.

It’s been an amazing intellectual journey for me and all of us. From the very beginning we had a strong set of fundamental principles and a strong underlying design—and for three decades we’ve been able to just keep building more and more on these foundations, creating what is by now an unprecedentedly vast system that has nevertheless maintained its unity, elegance and, frankly, modernity. In the early years we concentrated particularly on abstract areas such as mathematics. But over time we’ve dramatically expanded, taking ever larger steps and covering ever more kinds of computation and knowledge.

Each new version represents both a lot of new ideas and a lot of hard work. But more than that, it represents ever greater leverage achieved with our technology. Because one of our key principles is automation, and at every version we’re building on all the automation we’ve achieved before—in effect, we’ve got larger and larger building blocks that we’re able to use to go further and further more and more quickly. And of course what makes this possible is all that effort that I and others have put in over the years maintaining a coherent design for the whole system—so all those building blocks from every different area fit perfectly together.

With traditional approaches to software development, it would have taken a great many years to create what we’ve added in Version 11. And the fact that we can deliver Version 11 now is a direct reflection of the effectiveness of our technology, our principles and our methodology. And as I look at Version 11, it’s very satisfying to see how far we’ve come not only in what’s in the system, but also in how effectively we can develop it. Not to mention that all these directions we’ve been pursuing for so many years as part of the logical development of our system have now turned out to be exactly what’s needed for many of today’s most active areas of technology development.

For many years we called our core system Mathematica. But as we added new directions in knowledge and deployment, and expanded far beyond things related in any way to “math”, we decided to introduce the concept of the Wolfram Language to represent the core of everything we’re doing. And the Wolfram Language now defines the operation not only of Mathematica, but also of Wolfram Development Platform and Wolfram Programming Lab, as well as other products and platforms. And because all our software engineering is unified, today we’re able to release Version 11 of all our Wolfram-Language-based systems, both desktop and cloud.

OK, so what’s the big new thing in Version 11? Well, it’s not one big thing; it’s many big things. To give a sense of scale, there are 555 completely new functions that we’re adding in Version 11—representing a huge amount of new functionality (by comparison, Version 1 had a total of 551 functions altogether). And actually that function count is even an underrepresentation—because it doesn’t include the vast deepening of many existing functions.

The way we manage development, we’ve always got a portfolio of projects going on, from fairly small ones, to ones that may take five years or more. And indeed Version 11 contains the results of several five-year projects. We’re always keen to deliver the results of our development as quickly as possible to users, so we’ve actually had several intermediate releases since Version 10—and effectively Version 11 represents the combination of many completely new developments together with ones that we’ve already previewed in 10.1, 10.2, 10.3 and 10.4. (Many functions that were tagged as “Experimental” in 10.x releases are now in full production form in Version 11.0.)

The First Things You Notice…

When you first launch Version 11 on your desktop the first thing you’ll notice is that notebooks have a new look, with crisper fonts and tighter design. When you type code there are lots of new autocompletions that appear (it’s getting harder and harder to type the wrong thing), and when you type text there’s a new real-time spellchecker, that we’ll be continually updating to make sure it has the latest words included.

If your computer system is set to any of a dozen languages other than English, you’ll also immediately see something else: every function is automatically annotated with a “code caption” in the language you’ve set:

When you actually run code, you’ll notice that messages look different too—and, very helpfully for debugging, they let you immediately see what chain of functions was being called when the message was produced.

3D Printing

There are lots of big, meaty new areas in Version 11. But let’s jump right into one of them: 3D printing. I made my first 3D printout (which didn’t last long before disintegrating) back in 2002. And we’ve had the ability to export to STL for years. But what’s new and exciting in Version 11 is that we’ve built a complete pipeline that goes from creating 3D geometry to having it printed on your 3D printer (or through a printing service).

Often in the past I’ve wanted to take a 3D plot and just make a 3D print of it. And occasionally I’ve been lucky, and it’s been easy to do. But most of the time it’s a fiddly, complicated process. Because graphics that display on the screen don’t necessarily correspond to geometry that can actually be printed on a 3D printer. And it turns out to be a difficult problem of 3D computational geometry to conveniently set up or repair the geometry so it really works on a 3D printer. (Oh, and if you get it wrong, you could have plastic randomly squirting out of your printer.)

In Version 11 it’s finally realistic to take any 3D plot, and just 3D print it. Or you can get the structure of a molecule or the elevation around a mountain, and similarly just 3D print it. Over the years I’ve personally made many 3D printouts. But each one has been its own little adventure. But now, thanks to Version 11, 3D printouts of everything are easy. And now that I think about it, maybe I need a 3D printout of the growth of the Wolfram Language by area for my desk…

Machine Learning & Neural Networks

In a sense, Mathematica and the Wolfram Language have always been doing AI. Over the years we’ve certainly been pioneers in solving lots of “AI-ish” problems—from mathematical solving to automated aesthetics to natural language understanding. But back in Version 10 we also made a great step forward in machine learning—developing extremely automated core functions (Classify and Predict) for learning by example.

I have to say that I wasn’t sure how well these functions would do in practice. But actually it’s been amazing to see how well they work—and it’s been very satisfying to see so many of our users being able to incorporate machine learning into their work, just using the automation we’ve built, and without having to consult any machine learning experts.

In Version 11 we’ve made many steps forward in machine learning. We’ve now got clean ways not just to do classification and prediction, but also to do feature extraction, dimension reduction, clustering and so on. And we’ve also done a lot of training ourselves to deliver pre-trained machine-learning functions. Machine-learning training is an interesting new kind of development. At its core, it’s a curation process. It’s just that instead of, say, collecting data on movies, you’re collecting as many images as possible of different kinds of animals.

Built into Version 11 are now functions like ImageIdentify that identify over 10,000 different kinds of objects. And through the whole design of the system, it’s easy to take the features that have been learned, and immediately use those to train new image classifiers vastly more efficiently than before.

We’ve done a lot to automate today’s most common machine learning tasks. But it’s become clear in the past few years that an amazing number of new areas can now be tackled by modern machine learning methods, and particularly by using neural networks. It’s really an amazing episode in the history of science: the field of neural networks, that I’ve followed for almost 40 years, has gone from seeming basically hopeless to being one of the hottest fields around, with major new discoveries being made almost every week.

But, OK, if you want to get involved, what should you do? Yes, you can cobble things together with a range of low-level libraries. But in building Version 11 we set ourselves the goal of creating a streamlined symbolic way to set up and train neural networks—in which as much of what has to be done as possible has been automated. It’s all very new, but in Version 11 we’ve now got functions like NetGraph and NetChain, together with all sorts of “neural net special functions”, like DotPlusLayer and ConvolutionLayer. And with these functions it’s easy to take the latest networks and quickly get them set up in the Wolfram Language (recurrent networks didn’t quite make it into Version 11.0, but they’re coming soon).

Of course, what makes this all really work well is the integration into the rest of the Wolfram Language. The neural network is just a Graph object like any other graph. And inputs like images or text can immediately and automatically be processed using standard Wolfram Language capabilities into forms appropriate for neural network computation.

From their name, “neural networks” sound like they’re related to brains. But actually they’re perfectly general computational structures: they just correspond to complex combinations of simple functions. They’re not unrelated to the simple programs that I’ve spent so long studying—though they have the special feature that they’re set up to be easy to train from examples.

We’ve had traditional statistical data fitting and interpolation forever. But what’s new with neural networks is a vastly richer space of possible computational structures to fit data to, or to train. It’s been remarkable just in the past couple of years to see a sequence of fields revolutionized by this—and there will be more to come.

I’m hoping that we can accelerate this with Version 11. Because we’ve managed to make “neural net programming” really just another programming paradigm integrated with all the others in the Wolfram Language. Yes, it’s very efficient and can deal with huge training sets. But ultimately probably the most powerful thing is that it immediately fits in with everything else the Wolfram Language does. And even in Version 11 we’re already using it in many of our internal algorithms in areas like image, signal and text processing. It’s still early days in the history of “neural net programming”—but I’m excited for the Wolfram Language to play a central part in what’s coming.

Audio

OK, let’s turn to another new area of Version 11: audio. Our goal is to be able to handle any kind of data directly in the Wolfram Language. We’ve already got graphics and images and geometry and networks and formulas and lots else, all consistently represented as first-class symbolic structures in the language. And as of Version 11, we’ve now always got another type of first-class data: audio.

Audio is mostly complicated because it’s big. But in Version 11 we’ve got everything set up so it’s seamless to handle, say, an hour of audio directly in the Wolfram Language. Behind the scenes there’s all sorts of engineering that’s caching and streaming and so on. But it’s all automated—and in the language it’s just a simple Audio object. And that Audio object is immediately amenable to all the sophisticated signal processing and analysis that’s available in the Wolfram Language.

Bones, Foods, and the Universe…

The Wolfram Language is a knowledge-based language. Which means it’s got lots of knowledge—about both computation and the world—built into it. And these days the Wolfram Language covers thousands of domains of real-world knowledge—from countries to movies to companies to planets. There’s new data flowing into the central Wolfram Knowledgebase in the cloud all the time, and we’re carefully curating data on new things that exist in the world (who knew, for example, that there were new administrative divisions recently created in Austria?). Lots of this data is visible in Wolfram|Alpha (as well as intelligent assistants powered by it). But it’s in the Wolfram Language that the data really comes alive for full computation—and where all the effort we put into ensuring its alignment and consistency becomes evident.

We’re always working to expand the domains of knowledge covered by the Wolfram Language. And in Version 11 several domains that we’ve been working on for many years are finally now ready. One that’s been particularly difficult is anatomy data. But in Version 11 we’ve now got detailed 3D models of all the significant structures in the human body. So you can see how those complicated bones in the foot fit together. And you can do computations on them. Or 3D print them. And you can understand the network of arteries around the heart. I must say that as I’ve explored this, I’m more amazed than ever at the level of morphological complexity that exists in the human body. But as of Version 11, it’s now a domain where we can actually do computations. And there are perhaps-unexpected new functions like AnatomyPlot3D to support it. (There’s certainly more to be done, by the way: for example, our anatomy data is only for a single “average adult male”, and the joints can’t move, etc.)

A completely different domain of data now handled in the Wolfram Language is food. There’s a lot that’s complicated in this domain. First, there are issues of ontology. What is an apple? Well, there’s a generic apple, and there are also many specific types of apples. Then there are issues of defining amounts of things. A cup of strawberries. Three apples. A quarter pounder. It’s taken many years of work, but we’ve now got a very robust symbolic way to represent food—from which we can immediately compute nutritional properties and lots of other things.

Another area that’s also been long coming is historical country data. We’ve had very complete data on countries in modern times (typically from 1960 or 1970 on). But what about earlier history? What about Prussia? What about the Roman Empire? Well, in Version 11 we’ve finally got at least approximate border information for all serious country-like entities, throughout recorded history. So one can do computations about the rise and fall of empires right from within the Wolfram Language.

Talking of history, a small but very useful addition in Version 11 is historical word frequency data. Just ask WordFrequencyData for a time series, and you’ll be able to see how much people talked about “war”—or “turnips”—at different times in history. Almost every plot is a history lesson.

Another convenient function in Version 11 is WikipediaData, which immediately gives any Wikipedia entry (or various kinds of data it contains). There’s also WolframLanguageData, which gives computable data on the Wolfram Language itself—like the examples in its documentation, links between functions, and so on.

In many domains one’s mostly just dealing with static data (“what is the density of gold?”; “what was the population of London in 1959?”). But there are other domains where one’s not really interested in static data so much as data-backed computation. There are several new examples of this in Version 11. Like human mortality data (“what is the probability of dying between age X and Y?”), standard ocean data (“what is the pressure at a depth X?”), radioactive stopping power and human growth data—as well as data on the whole universe according to standard cosmological models.

Also new in Version 11 are WeatherForecastData and MathematicalFunctionData. Oh, as well as data on Pokémon and lots of other useful things.

Computing with Real-World Entities

One of the very powerful features of the Wolfram Language is its ability to compute directly with real-world entities. To the Wolfram Language, the US, or Russia, or a type of lizard are all just entities that can be manipulated as symbolic constructs using the overall symbolic paradigm of the language. Entities don’t directly have values; they’re just symbolic objects. But their properties can have values: [[USA]]["Population"] is 322 million.

But let’s say we don’t just want to take some entity (like the US) and find values of its properties. Let’s say instead we want to find what entities have certain properties with particular values. Like let’s say we want to find the 5 largest countries in the world by population. Well, in Version 11 there’s a new way to do this. Instead of specifying a particular explicit entity, we instead specify a computation that implicitly defines a class of entities. And so for example we can get a list of the 5 largest countries by population like this:

TakeLargest[5] is an operator form of a new function in Version 11 that gets the largest elements in a list. Implicit entities end up making a lot of use of operator forms—much like queries in Dataset. And in a sense they’re also making deep use of the symbolic character of the Wolfram Language—because they’re treating the functions that define them just like data.

The whole mechanism of entities and properties and implicit entities works for all the different types of entities that exist in the Wolfram Language. But as of Version 11, it’s not limited to built-in types of entities. There’s a new construct called EntityStore that lets you define your own types of entities, and specify their properties and values and so on—and then seamlessly use them in any computation.

Just as Dataset is a powerful hierarchical generalization of typical database concepts, so EntityStore is a kind of symbolic generalization of a typical relational database. And if you set up a sophisticated entity store, you can just use CloudDeploy to immediately deploy it to the cloud, so you can use it whenever you want.

Geo Everything

One aspect of “knowing about the real world” is knowing about geography. But the Wolfram Language doesn’t just have access to detailed geographic data (not only for Earth, but also for the Moon, Mars, even Pluto); it can also compute with this data. It’s got a huge collection of geo projections, all immediately computable, and all set up to support very careful and detailed geodesy. Remember spherical trigonometry? Well, the Wolfram Language doesn’t just assume the Earth is a sphere, but correctly computes distances and areas and so on, using the actual shape of the Earth.

When it comes to making maps, the Wolfram Language now has access not only to the street map of the world, but also to things like historical country borders, as well as at least low-resolution satellite imagery. And given the street map, there’s an important new class of computations that can be done: travel directions (and travel times) along streets from anywhere to anywhere.

Don’t Forget Calculus…

Version 11 has lots and lots of new capabilities across all areas of the Wolfram Language. But it’s also got lots of new capabilities in traditional Mathematica areas—like calculus. And back in earlier versions, what we’ve just added for calculus in Version 11 is big enough that it would have undoubtedly been the headline new feature of the version.

One example is differential eigensystems: being able to solve eigenvalue versions of both ODEs and PDEs. There’s a huge stack of algorithmic technology necessary to make this possible—and in fact we’ve been building towards it for more than 25 years. And what’s really important is that it’s general: it’s not something where one has to carefully set up some particular problem using elaborate knowledge of numerical analysis. It’s something where one just specifies the equations and their boundary conditions—and then the system automatically figures out how to solve them.

Back around 1976 I wrote a Fortran program to solve an eigenvalue version of the 1D Schrödinger equation for a particle physics problem I was studying. In 1981 I wrote C programs to do the same thing for some equations in relativistic quantum mechanics. I’ve been patiently waiting for the day when I can just type in these problems, and immediately get answers. And now with Version 11 it’s here.

Of course, what’s in Version 11 is much more powerful and more general. I was dealing with simple boundary conditions. But in Version 11 one can use the whole Wolfram Language geometry system—and all the data we have—to set up boundary conditions. So it’s easy to find the eigenmodes of a “drum” of any shape—like the shape of the US.

For something like that, there’s no choice but to do the computation numerically. Still, Version 11 will do differential eigensystem computations symbolically when it can. And Version 11 also adds some major new capabilities for general symbolic differential equations. In particular, we’ve had a large R&D project that’s now gotten us to the point where we can compute a symbolic solution to pretty much any symbolic PDE that would appear in any kind of textbook or the like.

Back in 1979 when I created a precursor to Mathematica I made a list of things I hoped we’d eventually be able to do. One of the things on that list was to solve integral equations. Well, I’m excited to be able to say that 37 years later, we’ve finally got the algorithmic technology stack to make this possible—and Version 11 introduces symbolic solutions to many classes of integro-differential equations.

There’s more in calculus too. Like Green’s functions for general equations in general domains. And, long awaited (at least by me): Mellin transforms. (They’ve been a favorite of mine ever since they were central to a 1977 particle physics paper of mine.)

It’s not classic calculus fare, but in Version 11 we’ve also added a lot of strength in what one might consider “modern calculus”—the aspects of calculus needed to support areas like machine learning. We’ve got more efficient and robust minimization, and we’ve also got sophisticated Bayesian minimization, suitable for things like unsupervised machine learning.

Education

Things like partial differential equations are sophisticated math, that happen to be very important in lots of practical applications in physics and engineering and so on. But what about more basic kinds of math, of the kind that’s for example relevant to high-school education? Well, for a long time Mathematica has covered that very thoroughly. But as our algorithmic technology stack has grown, there are a few new things that become possible, even for more elementary math.

One example new in Version 11 is full automatic handling of discontinuities, asymptotes, and so on in plots of functions. So now, for example, Tan[x] is plotted in the perfect high-school way, not joining –∞ and +∞. For Tan[x] that’s pretty simple to achieve. But there’s some seriously sophisticated algorithmic technology inside to handle this for more complicated functions.

And, by the way, another huge new thing in Version 11 is MathematicalFunctionData—computable access to 100,000 properties and relations about mathematical functions—in a sense encapsulating centuries of mathematical research and making it immediately available for computation.

We’ve been doing a lot recently using the Wolfram Language as a way to teach computational thinking at all levels. And among many other things, we’ve wanted to make sure that any computation that comes up—say in math—in elementary school education is really elementary to do in the Wolfram Language. And so we’ve got little functions like NumberExpand, which takes 123 and writes it as {100, 20, 3}. And we’ve also got RomanNumeral and so on.

And, partly as a tribute to the legacy of Logo, we’ve introduced AnglePath—a kind of industrial-scale version of “turtle graphics”, that happens to be useful not just for elementary education, but for serious simulations, say of different types of random walks.

Making Everything Fit Together

One of the central goals of the Wolfram Language is to have everything seamlessly work together. And in Version 11 there are some powerful new examples of this going on.

Time series, for example, now work directly with arithmetic. So you can take two air pressure time series, and just subtract them. Of course, this would be easy if all the time points in the series lined up. But in Version 11 they don’t have to: the Wolfram Language automatically handles arbitrarily irregular time series.

Another example concerns units. In Version 11, statistical distributions now work seamlessly with units. So a normal distribution can have not just a variance of 2.5, but a variance of 2.5 meters. And all computations and unit conversions are handled completely automatically.

Geometry and geometric regions have also been seamlessly integrated into more parts of the system. Solvers that used to just take variables can now be given arbitrary regions to operate over. Another connection is between images and regions: ImageMesh now takes any image and constructs a geometric mesh from it. So, for example, you can do serious computational geometry with your favorite cat picture if you want.

One final example: random objects. RandomInteger and RandomReal are old functions. Version 8 introduced RandomVariate for picking random objects from arbitrary symbolic probability distributions. Then in Version 9 came RandomFunction, for generating functions from random processes. But now in Version 11 there’s more randomness. There’s RandomPoint, which picks a random point in any geometric region. And there’s also RandomEntity that picks a random entity, as well as RandomWord—that’s useful for natural language processing research, as well as being a nice way to test your vocabulary in lots of languages… And finally, in Version 11 there’s a whole major new mathematical area of randomness: random matrices—implemented with all the depth and completeness that we’ve made a hallmark of Mathematica and the Wolfram Language.

Visualization

One of the long-term achievements of Mathematica and the Wolfram Language has been that they’ve made visualization a routine part of everyday work. Our goal has always been to make it as automatic as possible to visualize as much as possible. And Version 11 now makes a whole collection of new things automatic to visualize.

There are very flexible word clouds that let one visualize text and collections of strings. There are timeline plots for visualizing events in time. There are audio plots that immediately visualize short and long pieces of audio. There are dendrograms that use machine learning methods to show hierarchical clustering of images, text, or any other kind of data. There are geo histograms to show geographic density. There’s a TextStructure function that diagrams the grammar of English sentences. And there are anatomy plots, to show features in the human body (making use of symbolic specifications, since there aren’t any explicit coordinates).

What other kinds of things are there to visualize? Well, one thing I’ve ended up visualizing a lot (especially in my efforts in basic science) are the rules for simple programs like cellular automata. And in Version 11 we’ve added RulePlot for automatically visualizing rules in many different styles.

Another longstanding visualization challenge has been how to automatically visualize 3D distributions of data. The issue tends to be that it’s hard to “see into” the 3D volume. But in Version 11 we’ve got a bunch of functions that solve this in different ways, often by making slices at positions that are defined by our geometry system.

In the quest for automation in visualization, another big area is labeling. And in Version 11 we’ve added Callout to make it possible to specify callouts for points, lines and regions (we’ve already got legends and tooltips and so on). There’s a trivial way to do callouts: just always put them (say) on the left. But that’d be really bad in practice, because lots of callouts could end up colliding. And what Version 11 does instead is something much more sophisticated, involving algorithmically laying out callouts to optimally achieve aesthetic and communication goals.

From Strings to Text…

Mathematica and the Wolfram Language have always been able to handle character strings. And in Version 10 there was a huge step forward with the introduction of Interpreter—in which we took the natural language understanding breakthroughs we made for Wolfram|Alpha, and applied them to interpreting strings in hundreds of domains. Well, in Version 11 we’re taking another big step—providing a variety of functions for large-scale natural language processing and text manipulation.

There are functions like TextWords and TextSentences for breaking text into words and sentences. (It takes fancy machine learning to do a good job, and not, for example, to be confused by things like the periods in “St. John’s St.”) Then there are functions like TextCases, which lets one automatically pick out different natural-language classes, like countries or dates, or, for that matter, nouns or verbs.

It’s pretty interesting being able to treat words as data. WordList gives lists of different kinds of words. WordDefinition gives definitions.

Then there are multilingual capabilities. Alphabet gives pretty much every kind of alphabet; Transliterate transliterates between writing systems. And WordTranslation gives translations of words into about 200 languages. Great raw material for all sorts of linguistics investigations.

The Most Modern Systems Programming

The Wolfram Language is arguably the highest-level language that’s ever been created. But in Version 11 we’ve added a bunch of capabilities for “reaching all the way down” to the lowest level of computer systems. First of all, there’s ByteArray that can store and manipulate raw sequences of bytes. Then there are functions that deal with raw networks, like PingTime and SocketConnect.

There’s a new framework for publish-subscribe “channels”. You can create a channel, then either the Wolfram Language or an external system can send it data, and you can set up a “listener” that will do something in your Wolfram Language session whenever the data arrives. There’s a lot that can be built with this setup, whether it’s connecting to external services and devices, handling notifications and third-party authentication—or creating your very own chat system.

Something else new in Version 11 is built-in cryptography. It’s a very clean symbolic framework that lets you set up pretty much whatever protocol you want, using public or private key systems.

What about interacting with the web? The symbolic character of the Wolfram Language is again very powerful here. Because for example it lets one have HTTPRequest and HTTPResponse as symbolic structures. And it also lets one have functions like URLSubmit, with symbolically defined handler functions for callbacks from asynchronous URL submission. There’s even now a CookieFunction, for symbolic handling of cookies.

Yes, one can do systems programming in pretty much any language—or even for example in a shell. But what I’ve found is that doing it in the Wolfram Language is incredibly more powerful. Let’s say you’re exploring the performance of a computer system. Well, first of all, everything you’re doing is kept nicely in a notebook, where you can add comments, etc. Then—very importantly—everything you do can be immediately visualized. Or you can apply machine learning, or whatever. Want to study network performance? Use PingTime to generate a list of ping times; then immediately make a histogram, correlate with other data, or whatever.

Something else we’re adding in Version 11 is FileSystemMap: being able to treat a file system like a collection of nested lists (or associations) and then mapping any function over it. So for example you can take a directory full of images, and use FileSystemMap to apply image processing to all of them.

Oh, and another thing: Version 11 also includes, though it’s still tagged as experimental, a full industrial-strength system for searching text documents, both locally and in the cloud.

Building Things on the Web

An incredibly powerful feature of the Wolfram Language is that it runs not only on the desktop but also in the cloud. And in Version 11 there are lots of new capabilities that use the cloud, for example to create things on the web.

Let’s start with the fairly straightforward stuff. CloudDeploy[FormFunction[...]] lets you immediately create a form-based app on the web. But now it’s easy to make the form even more sophisticated. There are lots of new types of “smart fields” that automatically use natural language understanding to interpret your input. There are new constructs, like RepeatingElement and CompoundElement, that automatically set up fields to get input for lists and associations. And there’s a whole new Programmable Linguistic Interface that lets you define your own grammar to extend the natural language understanding that’s already built into the Wolfram Language.

The forms you specify symbolically in the Wolfram Language can be quite sophisticated—with multiple pages and lots of interdependencies and formatting. But ultimately they’re still just forms where you set up your input, then submit it. Version 11 introduces the new AskFunction framework which lets you set up more complex interactions—like back-and-forth dialogs in which you “interview” the user to get data. In the Wolfram Language, the whole process is specified by a symbolic structure—which CloudDeploy then makes immediately active on the web.

It’s a goal of the Wolfram Language to make it easy to build complex things on the web. In Version 11, we’ve added FormPage to let you build a “recycling form” (like on wolframalpha.com), and GalleryView to let you take a list of assets in the Wolfram Language, and immediately deploy them as a “gallery” on the web (like in demonstrations.wolfram.com).

If you want to operate at a lower level, there are lots of new functions, like URLDispatcher and GenerateHTTPResponse, that let you determine exactly how web requests will be handled by things you set up in the cloud.

Also new in Version 11 are functions like CloudPublish and CloudShare that let you control access to things you put in the cloud from the Wolfram Language. A small but I think important new feature is SourceLink, which lets you automatically link from, say, a graphic that you deploy in the cloud back to the notebook (also in the cloud) in which it was created. I think this will be a great tool for “data-backed publication”—in which every picture you see in a paper, for example, links back to what created it. (Inside our company, I’m also insisting that automated reports we generate—from the Wolfram Language of course—include source links, so I can always get the raw data and analyze it myself or whatever.)

Data Into the Cloud

Already there experimentally in Version 10—but now fully supported in Version 11—is the whole Wolfram Data Drop mechanism, which lets you accumulate data from anywhere into the Wolfram Cloud. I have to say I think I underestimated the breadth of usefulness of the Wolfram Data Drop. I thought it would be used primarily to store data from sensors and the like. And, yes, there are lots of applications along these lines. But what I’ve found is that the Data Drop is incredibly useful purely inside the Wolfram Language. Let’s say you’ve got a web form that’s running in the Wolfram Language. You might process each request—but then throw the result into a databin in the Wolfram Data Drop so you can analyze them all together.

Wolfram Data Drop is basically set up to accumulate time series of data. In Version 11 another way to store data in the cloud is CloudExpression. You can put any Wolfram Language expression into a cloud expression, and it’ll be persistently stored there, with each part being extracted or set by all the usual operations (like Part and AppendTo) that one could use on a symbol in a Wolfram Language session. CloudExpression is a great way to store structured data where one’s continually modifying parts, but one wants all the data to be persistent in the cloud.

Things you store in the cloud are immediately persistent. In Version 11, there’s also LocalObject—which is the local analog of CloudObject—and provides persistent local storage on your machine. LocalCache is a seamless way of ensuring that things you’re using are cached in the local object store.

Connecting Everywhere

In the Wolfram Language we curate lots of data that we include directly in the knowledgebase—but we also curate ways to access more data, such as external APIs. Version 11 includes many new connections, for example to Flickr, Reddit, MailChimp, SurveyMonkey, SeatGeek, ArXiv and more.

The Wolfram Language is also an extremely powerful way to deploy your own APIs. And in Version 11 there’s an expanding set of authentication mechanisms that are supported for APIs—for example PermissionsKey for giving appids. CloudLoggingData also now gives what can be extremely detailed data about how any API or other cloud object you have is being accessed.

An API that you call on the web basically gets data passed to it through the URL that it’s given. In Version 11 we have a new kind of API-like construct that operates not through the web and URLs, but through email. MailReceiverFunction is like APIFunction, except when it’s deployed it defines an email address, and then any mail that’s sent to that email address gets fed to the code in the mail receiver function. MailReceiverFunction lets you get very detailed in separating out different parts of a mail message and its headers—and then lets you apply any Wolfram Language function you want—so you can do arbitrarily sophisticated automation in processing email. And particularly for someone like me who gets a huge amount of email from humans as well as automated systems, this is a pretty nice thing.

WolframScript

You can access the Wolfram Language through a notebook, on the desktop or in the cloud. You can access it through a scheduled task in the cloud, or through an API or a mail receiver function. It’s always been possible to run the Wolfram Language from a command line too, but in Version 11 there’s a powerful new way to do that, using WolframScript.

The idea of WolframScript is to provide a very simple but flexible interface to the Wolfram Language. WolframScript lets you run on a local Wolfram Engine on your computer—or just by saying -cloud it lets you run in the cloud. It lets you run code from a file or directly from the command line. And it lets you get back results in any format—including text or images or sounds or PDF or CDF or whatever. And in the usual Unix way, it lets you use #!wolframscript to create a script that can be called standalone and will run with WolframScript.

There’s more too. You can set up WolframScript to operate like a Wolfram Language FormFunction—pulling in arguments of whatever types you specify (and doing interpretation when needed). And you can also use WolframScript to call an API you’ve already defined in the cloud.

In our own company, there are lots of places where we’re using the Wolfram Language as part of some large and often distributed system. WolframScript provides a very clean way to just “throw in a Wolfram Language component” anywhere you want.

The Core Language

I’ve talked about all sorts of things that broaden and deepen the algorithmic capabilities of Mathematica and the Wolfram Language. But what about the structure of the core Wolfram Language itself? Of course, we’re committed to always maintaining compatibility (and I’m happy to say that all our attention to design on an ongoing basis tends to make this rather easy). But we also want to progressively strengthen and polish the language.

In natural languages, one process of evolution tends to be the construction of new words from common idioms. And we’re doing essentially the same thing in the Wolfram Language. We’ve made an active effort to study what “lumps of repeated computational work” appear most frequently across lots of Wolfram Language code. Then—assuming we can come up with a good name for a particular lump of computational work—we add it as a new function.

In the early days of Mathematica I used to take the point of view that if there were functions that let you do something using an idiom, then that was fine. But what I realized is that if an idiom is compressed into a single function whose name communicates clearly what it’s for, then one gets code that’s easier to read. And coupled with the convenience of not have to reconstruct an idiom many times, this justifies having a new function.

For the last several years, two initiatives we’d had within our company are Incremental Language Development (ILD), and Language Consistency & Completeness (LCC). The idea of ILD is to do things like introduce functions that are equivalent to common idioms. The idea of LCC is to do things like make sure that anything—like pattern matching, or units, or symbolic URLs—is supported wherever it makes sense across the system.

So, for example, a typical ILD addition in Version 11 is the function MinMax that returns min and max in a list (it’s amazing how many fiddly applications of Map that saves). A typical LCC addition is support for pattern matching in associations.

Between ILD and LCC there are lots of additions to the core language in Version 11. Functions like Cases have been extended to SequenceCases—looking for sequences in a list instead of individual elements. There’s also now SequenceFoldList, which is like FoldList, except it can “look back” to a sequence of elements of any length. In a similar vein, there’s FoldPairList, which generalizes FoldList by allowing the result “returned” at each step to be different from the result that’s passed on in the folding process. (This might sound abstract, and at some level it is—but this is a very useful operation whenever one wants to maintain a separate internal state while sequentially ingesting data.)

Another new construct that might at first sound weird is Nothing. Despite its name, Nothing does something very useful: whenever it appears in a list, it’s immediately removed. Which means, for example, that to get rid of something in a list, you just have to replace it by Nothing.

There are lots of little conveniences we’ve added in Version 11. First, for example, now has a second argument, that says what to give if there’s no first element—and avoids having to include an If for that case. There’s also a nice general mechanism for things like this: UpTo. You can say Take[list,UpTo[4]] to get up to 4 elements in list, or however many elements there happen to be. UpTo is supported in lots of places—and it simplifies lots of code.

Another little convenience is Echo. When you’re trying to tell what’s going on inside a piece of code, you sometimes want to print some intermediate result. Echo is a function that prints, then returns what it printed—so you can sprinkle it in your code without changing what the code does.

It’s hard to believe there are useful basic list operations still to add, but Version 11 has a few. Subdivide is like Range except it subdivides the range into equal parts. TakeLargest and related functions generalize Max etc. to give not just the largest, but the n largest elements in a list.

There’s a function Groupings, which I’d been thinking about for years but only now figured out a good design for—and which effectively generates all possible trees formed with certain binary or other combiners (“what numbers can you get by combining a list of 1s in all possible ways with Plus and Times?”).

There are nips and tucks to the language in all sorts of places. Like in Table, for example, where you can say Table[x,n] rather than needing to say Table[x,{n}]. And in general there are lots and lots of things that make the core of Version 11 of the Wolfram Language just a little smoother, nicer and more elegant to use.

And There’s Much More

This has been a long blog post. But I’m not even close to having covered everything that’s new in Version 11. There’s more information on the web. Check out the featured new areas, or the information for existing users or the summary of new features. (See also the list of new features specifically from Version 10.4 to 11.0.)

But most of all, get into using Version 11! If you want quick (and free) exposure to it, try it in the Wolfram Open Cloud. Or just start using Mathematica 11 or Version 11 of any other products based on the Wolfram Language.

I’ve been using test versions of Version 11 for some time now—and to me Version 10 already looks and feels very “old fashioned”, lacking those nice new interface features and all the new functionality and little conveniences. I’m really pleased with the way Version 11 has turned out. It’s yet another big step in what’s already been a 30-year journey of Mathematica and the Wolfram Language. And I’m excited to see all the wonderful things that all sorts of people around the world will now be able to do for first time with Mathematica 11 and the other Version 11 Wolfram Language products.

To comment, please visit the copy of this post at the Wolfram Blog »