This week I’m giving a talk at a conference on Mathematics and Computation in Music (MCM 2011)… so I decided to collect some of my thoughts on such topics…

How difficult is it to generate human-like music? To pass the analog of the Turing test for music?

Though music typically has a certain formal structure—as the Pythagoreans noted 2500 years ago—it seems at its core somehow fundamentally human: a reflection of raw creativity that is almost a defining characteristic of human capabilities.

But what is that creativity? Is it something that requires the whole history of our biological and cultural evolution? Or can it exist just as well in systems that have nothing directly to do with humans?

In my work on A New Kind of Science, I studied the computational universe of possible programs—and found that even very simple programs can show amazingly rich and complex behavior, on a par, for example, with what one sees in nature. And through my Principle of Computational Equivalence I came to believe that there can be nothing that fundamentally distinguishes our human capabilities from all sorts of processes that occur in nature—or in very simple programs.

But what about music? Some people used their belief that “no simple program will ever create great music” to argue that there must be something wrong with my Principle of Computational Equivalence.

So I became curious: is there really something special and human about music? Or can it in fact be created perfectly well in an automatic, computational way?

In 2003, after my decade as a recluse working on A New Kind of Science, I started to be out and about more—and kept on having the mundane problem that my cellphone had the same ringtone as lots of others. So I thought: if distinctive original music could in fact be generated automatically, then one could just “mass customize” cellphone ringtones, and everyone could have their own.

A little while later we decided to try some experiments—and see just what might be possible in creating music from programs.

There’s a long history of attempts to produce music from rules. Most of it seems either too robotic or too random. But the discoveries I made in A New Kind of Science seemed to offer new possibilities—because they showed that even with the rules of a simple program, it was possible to produce the kind of richness and complexity that, for example, we see and admire in nature.

We started with the most obvious experiment: take the cellular automata that I had studied so much, and use slices of the patterns they generate to form musical scores. I had no real idea what the result of this would be. And certainly some cellular automata with simple patterns of behavior produced completely boring music.  But somewhat to my surprise, one really didn’t have to go far in the computational universe of possible cellular automata before one started to find remarkable rich and pleasing pieces of music.

The fact that there was always just a simple program underneath gave a certain inevitable logic to the music. But the key point from the science was that even though the underlying program was simple, the pattern it produced could be rich and complex.

But would it be aesthetic? In the visual domain, I had known for a long time that cellular automata could produce pleasing and interesting patterns. And in a sense, given my scientific discoveries, this wasn’t surprising. Because I knew that cellular automata could capture the essence of many processes in nature. And insofar as we find nature to be aesthetic, so also this should be true of cellular automata.

But whereas nature just uses a few particular kinds of rules, the complete universe of cellular automata is infinite. In a sense, that computational universe generalizes our actual universe. It keeps the essential mechanisms, but allows an infinite diversity of variations—each with aesthetics that generalize the aesthetics of the natural world.

Ever since the beginning of the 1990s, Mathematica had supported sound generation. And with its symbolic language, Mathematica provided the ideal platform for us to implement our algorithms and start generating music. The results greatly exceeded even our most positive expectations. We used ideas from music theory to take raw cellular automaton creations and “dress them”—and very soon we were producing orchestrated musical pieces that sounded remarkably good.

Around our offices people would sometime overhear what was being produced—and stop to ask “What song are you listening to?” We were making music that was good enough that people assumed it must have the usual human origins: we had succeeded in passing the analog of the Turing test for music.

Well, we soon built a website that we called WolframTones. And all sorts of people started using it. I must say that I thought it was an interesting intellectual experiment—and perhaps a good way to make simple ringtones—but not something that one would take very seriously from a musical point of view.

But I was wrong. Pretty quickly all sorts of serious composers started using the site. They would tell us that they found it useful as a source of ideas—as a source of creative seeds for their compositions. In a sense this was bizarre. We had started unsure of whether computers could achieve anything close to human creativity. Yet now skilled humans were coming to our automated system to seek what we might have thought was that uniquely human thing: creative inspiration.

To me, this was a nice validation of the Principle of Computational Equivalence. As one researcher put it: “Once one’s heard the music they produce, simple programs seem a lot more like us”.

Out in the computational universe, each program in effect defines its own artificial world—whose sounds and logic we get to hear in the music it produces. Some of those worlds are boring, arid places that yield dull, monotonous music. Others are rife with randomness and noise. But every hundred or thousand programs, one finds something wonderful: rich, textured, sometimes familiar, sometimes exotic, musical form.

On the WolframTones website we let people press buttons to go and search at random for music that fits into heuristics we’ve defined for various standard musical genres. We also let people incrementally modify the rules for a piece of music—in effect applying artificial selection to evolve variations they want. And when one uses WolframTones, it feels a bit like doing nature photography. We explore the computational universe to find those corners—those particular programs—that have the significance or aesthetics that we want.

The WolframTones website went live on September 16, 2005. And ever since then it’s just been out there on the web, running Mathematica, and creating music. I must admit that I hadn’t looked at its logs for quite a while. But doing that now, I discover that it has been used tens of millions of times—creating tens of millions of musical compositions.

By the standards, for example, of Wolfram|Alpha usage, that’s nothing. But by the standards of musical composition, it’s huge. iTunes now has about 14 million pieces on it—representing most of the published musical output of our species.  But in just a few short years, WolframTones has created more compositions than that. By pure computation, it has in a sense surpassed our species in musical output, single-handedly creating more original music than in the whole history of music before it.

To allow instant output, the website encodes music using MIDI (something that the Mathematica language now supports in a direct symbolic way). Many arrangements of WolframTones output as MP3 have been made. And in a peculiar reversal of roles, I went to a recital a few years ago where human performers were playing on violins a piece that had been entirely created using WolframTones methods.

Can simple programs create a complete symphony? A WolframTones composition explores for perhaps a minute the story of some particular computational world. My experience is that to create a longer piece—that tells a bigger story—seems to require higher-level structure. But there is nothing wrong with having a simple program provide that structure. The overarching story it tells can be perfectly compelling, just like so many stories that play out under the aegis of natural laws in the natural world.

But just how much can come from how little? What is the shortest program that makes an interesting musical piece?

It’s easy to start constructing Mathematica programs.

Sound[SoundNote[DeleteCases[3 Range[21] Reverse[#], 0] - 24, .1] & /@ Transpose[CellularAutomaton[90, {{1}, 0}, 20]]]

Sound[SoundNote[#, 1/6, "Warm"] & /@ (Pick[{0, 5, 9, 12, 16, 21}, #, 1] & /@ CellularAutomaton[30, {{1, 0, 0, 0, 0, 0}, 0}, 13, {13, 5}])]

And we’re planning to do a competition to see how good this can get, especially using all the modern algorithmic tools—like image processing, for example—that exist in Mathematica. But ultimately in such a quest we can’t rely on human creativity alone. We have, in effect, to automate this creativity—going beyond what humans have imagined, and instead just exploring the computational universe, and plucking from it the ideas and programs we want.

In creating music we can operate at the level of notes, or collections of notes—or even sound waveforms, generalizing the ways of constructing pleasing waveforms that physical musical devices (or their synthesized direct analogs) have traditionally used.

Of course, creativity from the computational universe is not limited to music. There’s been quite a lot of investigation, for example, in the visual arts, and in architecture. Can we create a building from a single, simple rule? If we can, the building will necessarily have a certain logic to its structure, that will allow humans to learn and be comfortable with it.

Can we really appreciate music or other forms that have been created automatically? Or do we always need a story that links what we see into the whole fabric of human culture? Once again, our appreciation of nature makes it clear that no human story is needed. Instead, what seems to be necessary is a connection to a certain overarching logic, which in a sense is precisely what the whole concept of the computational universe provides.

When I look at Wolfram|Alpha, I’m pleased at how much of systematic human knowledge we’re being able to capture, and make computable. A new frontier is to capture not just knowledge, but also creativity. To be able, for example, to go from a goal, and creatively work out how to achieve it. Music exposes us to a rather pure form of creativity—and what we have learned, as the Principle of Computational Equivalence might suggest, is that even in this domain, ideas like WolframTones do remarkably well at achieving creative output.

We’re going to be able to do another level of automation—in a sense dramatically broadening access to creativity, and no doubt enabling all sorts of fascinating new possibilities.