It’s Just Adding One Word at a Time

That ChatGPT can automatically generate something that reads even superficially like human-written text is remarkable, and unexpected. But how does it do it? And why does it work? My purpose here is to give a rough outline of what’s going on inside ChatGPT—and then to explore why it is that it can do so well in producing what we might consider to be meaningful text. I should say at the outset that I’m going to focus on the big picture of what’s going on—and while I’ll mention some engineering details, I won’t get deeply into them. (And the essence of what I’ll say applies just as well to other current “large language models” [LLMs] as to ChatGPT.)

The first thing to explain is that what ChatGPT is always fundamentally trying to do is to produce a “reasonable continuation” of whatever text it’s got so far, where by “reasonable” we mean “what one might expect someone to write after seeing what people have written on billions of webpages, etc.” Continue reading

The Mystery of the Second Law

Entropy increases. Mechanical work irreversibly turns into heat. The Second Law of thermodynamics is considered one of the great general principles of physical science. But 150 years after it was first introduced, there’s still something deeply mysterious about the Second Law. It almost seems like it’s going to be “provably true”. But one never quite gets there; it always seems to need something extra. Sometimes textbooks will gloss over everything; sometimes they’ll give some kind of “common-sense-but-outside-of-physics argument”. But the mystery of the Second Law has never gone away.

Why does the Second Law work? And does it even in fact always work, or is it actually sometimes violated? What does it really depend on? What would be needed to “prove it”?

For me personally the quest to understand the Second Law has been no less than a 50-year story. But back in the 1980s, as I began to explore the computational universe of simple programs, I discovered a fundamental phenomenon that was immediately reminiscent of the Second Law. And in the 1990s I started to map out just how this phenomenon might finally be able to demystify the Second Law. But it is only now—with ideas that have emerged from our Physics Project—that I think I can pull all the pieces together and finally be able to construct a proper framework to explain why—and to what extent—the Second Law is true. Continue reading

When I Was 12 Years Old…

I’ve been trying to understand the Second Law now for a bit more than 50 years.

It all started when I was 12 years old. Building on an earlier interest in space and spacecraft, I’d gotten very interested in physics, and was trying to read everything I could about it. There were several shelves of physics books at the local bookstore. But what I coveted most was the largest physics book collection there: a series of five plushly illustrated college textbooks. And as a kind of graduation gift when I finished (British) elementary school in June 1972 I arranged to get those books. And here they are, still on my bookshelf today, just a little faded, more than half a century later:

Continue reading

The Basic Arc of the Story

As I’ve explained elsewhere, I think I now finally understand the Second Law of thermodynamics. But it’s a new understanding, and to get to it I’ve had to overcome a certain amount of conventional wisdom about the Second Law that I at least have long taken for granted. And to check myself I’ve been keen to know just where this conventional wisdom came from, how it’s been validated, and what might have made it go astray.

And from this I’ve been led into a rather detailed examination of the origins and history of thermodynamics. All in all, it’s a fascinating story, that both explains what’s been believed about thermodynamics, and provides some powerful examples of the complicated dynamics of the development and acceptance of ideas. Continue reading

ChatGPT and Wolfram|Alpha

It’s always amazing when things suddenly “just work”. It happened to us with Wolfram|Alpha back in 2009. It happened with our Physics Project in 2020. And it’s happening now with OpenAI’s ChatGPT. Continue reading

Delivering from Our R&D Pipeline

In 2020 it was Versions 12.1 and 12.2; in 2021 Versions 12.3 and 13.0. In late June this year it was Version 13.1. And now we’re releasing Version 13.2. We continue to have a huge pipeline of R&D, some short term, some medium term, some long term (like decade-plus). Our goal is to deliver timely snapshots of where we’re at—so people can start using what we’ve built as quickly as possible. Continue reading

::[ ]

Threaded

MapApply

FeatureImpactPlot

IntegrateChangeVariables

ReconstructionMesh

VideoCapture

RootTree

ImplicitD

“ExprStruct”

ChemicalConvert

UniqueElements

ReplaceAt

TypeEvaluate

TreeLeafCount

QuestionSelector

IncludeSingularSolutions

KernelConfiguration

PhongShading

PatternReaction

FractionalD

OrdinalScale

DropShadowing

UnlabeledTree

FeatureValueDependencyPlot

CaputoD

BlockDiagonalMatrix

ModelPredictiveController

GeodesicPolyhedron

InertEvaluate

Until

TernaryListPlot

FromRawPointer

VideoScreenCapture

PermutationMatrix

FeatureValueImpactPlot

ApplyReaction

GraphJoin

TorusGraph

DSolveChangeVariables

CumulativeFeatureImpactPlot

TreeElementShape

LibraryFunctionDeclaration

SymmetricDifference

FlatShading

CircularArcThrough

GouraudShading

DiscreteInputOutputModel

EstimatedPointNormals

GraphProduct

NominalScale

QuestionGenerator

GraphSum

BuckyballGraph

ChemicalInstance

InertExpression

UpperTriangularMatrix

CreateTypeInstance

PlaybackSettings

DeleteElements

BlockUpperTriangularMatrix

CompilerCallback

BlockLowerTriangularMatrix

ToRawPointer

LocalEvaluate

LowerTriangularMatrix

UnmanageObject

TypeOf

DeleteAdjacentDuplicates

InfiniteLineThrough

$DefaultLocalKernels

TernaryPlotCorners

ForAllType

SameAs

ResidueSum

CompiledExpressionDeclaration

Cast

ParentEdgeStyle

TreeElementLabelFunction

TreeElementSizeFunction

ParentEdgeLabelStyle

ParentEdgeLabel

TreeElementSize

TreeElementLabelStyle

TreeElementCoordinates

VideoTransparency

TreeElementShapeFunction

ParentEdgeShapeFunction

ParentEdgeLabelFunction

TreeElementStyleFunction

ParentEdgeStyleFunction

🙀

The Epic Continues…

Last week it was 34 years since the original launch of Mathematica and what’s now the Wolfram Language. And through all those years we’ve energetically continued building further and further, adding ever more capabilities, and steadily extending the domain of the computational paradigm.

In recent years we’ve established something of a rhythm, delivering the fruits of our development efforts roughly twice a year. We released Version 13.0 on December 13, 2021. And now, roughly six months later, we’re releasing Version 13.1. As usual, even though it’s a “.1” release, it’s got a lot of new (and updated) functionality, some of which we’ve worked on for many years but finally now brought to fruition. Continue reading

The Nature of Alien Intelligence

“We’re going to launch lots of tiny spacecraft into interstellar space, have them discover alien intelligence, then bring back its technology to advance human technology by a million years”. I’ve heard some pretty wacky startup pitches over the years, but this might possibly be the all-time winner.

But as I thought about it, I realized that beyond the “absurdly extreme moonshot” character of this pitch, there’s some science that I’ve done that makes it clear that it’s also fundamentally philosophically confused. The nature of the confusion is interesting, however, and untangling it will give us an opportunity to illuminate some deep features of both intelligence and technology—and in the end suggest a way to think about the long-term trajectory of the very concept of technology and its relation to our universe.

Let’s start with a scenario. Let’s say one of the little spacecraft comes across a planet where it sees complicated swirling patterns:

&#10005

Continue reading

Humanizing Multicomputational Processes

Multicomputation is one of the core ideas of the Wolfram Physics Project—and in particular is at the heart of our emerging understanding of quantum mechanics. But how can one get an intuition for what is initially the rather abstract idea of multicomputation? A good approach, I believe, is to see it in action in familiar systems and situations. And I explore here what seems like a particularly good example: games and puzzles. Continue reading

From the Foundations Laid by A New Kind of Science

When A New Kind of Science was published twenty years ago I thought what it had to say was important. But what’s become increasingly clear—particularly in the last few years—is that it’s actually even much more important than I ever imagined. My original goal in A New Kind of Science was to take a step beyond the mathematical paradigm that had defined the state of the art in science for three centuries—and to introduce a new paradigm based on computation and on the exploration of the computational universe of possible programs. And already in A New Kind of Science one can see that there’s immense richness to what can be done with this new paradigm. Continue reading