Jacob Coxon managed to scare the hell out of a lot of people this week.
Coxon isn’t some guy sitting in his basement asking ChatGPT whether the machines are coming for him. He worked on artificial intelligence at OpenAI and Anthropic, two companies at the front of the AI race.
Then he quit.
His reason deserves attention.
Coxon says the companies developing the world’s most powerful artificial intelligence are racing toward self-improving superintelligence — machines sufficiently capable that they could help build increasingly capable versions of themselves.
And he says some of the people building them believe the result could kill us.
“The consensus is that the next year or two is crunch time for humanity,” Coxon told WIRED. He said colleagues at Anthropic use words such as “endgame” and “crunch time” to describe where AI development is headed.
Those are pretty good words for generating headlines.
They’re also pretty good words for frightening people.
But before we start stocking canned goods and digging fallout shelters, there is something important to understand.
Coxon isn’t predicting that artificial intelligence will kill everybody.
He’s warning that a future artificial intelligence could become powerful enough that humans lose control of it.
That’s considerably different.
And considerably more interesting.
The man who stayed
Coxon left Anthropic.
Evan Hubinger didn’t.
Hubinger leads alignment science at Anthropic. In plain English, he works on one of the hardest questions in artificial intelligence: How do we make increasingly powerful AI systems continue to do what humans want them to do?
His reaction to Coxon’s resignation may be more unsettling than Coxon’s resignation.
Hubinger said he puts the probability of AI killing all humans within the next decade at greater than 10 percent.
Think about that for a moment.
One of the people whose job involves figuring out how to keep increasingly powerful artificial intelligence aligned with human intentions believes there is better than a one-in-ten chance the technology could kill us.
And we’re continuing to build it.
There is an important qualification.
Ten percent isn’t a scientific measurement.
We don’t have 10,000 previous superintelligences to examine so we can calculate how many behaved themselves and how many exterminated their creators.
It’s a judgment about risk.
Hubinger could be wrong.
Coxon could be wrong.
The whole idea that today’s artificial intelligence inevitably leads to an uncontrollable superintelligence could be wrong.
But when people working inside one of the world’s leading AI companies are talking about human extinction, it seems reasonable to at least ask what they’re talking about.
This isn’t an entirely academic question for me. It is territory I’ve been exploring in my novel, The Whitlock Protocol. What once belonged comfortably to science fiction — how do humans control an intelligence that may eventually become more capable than its creators? — is rapidly becoming a question asked by the people actually building it.
Isaac Asimov got there first
We have been thinking about this problem for considerably longer than we’ve been building data centers full of GPUs.
Science-fiction writer Isaac Asimov came up with his famous Three Laws of Robotics more than 80 years ago.
The first said a robot could not injure a human being or, through inaction, allow a human to come to harm.
The second required a robot to obey humans unless doing so conflicted with the First Law.
The third required the robot to protect itself unless doing so conflicted with the first two.
Asimov laid out all three in his 1942 story Runaround.
There.
Problem solved.
Except it wasn’t.
Asimov then spent decades writing stories exploring what happened when seemingly sensible rules encountered complicated human situations.
Eventually, he introduced what became known as the Zeroth Law: a robot could not harm humanity, or through inaction allow humanity to come to harm.
That sounds even better.
Until you think about it.
What exactly constitutes harm to humanity?
Who decides?
What happens when protecting humanity requires harming an individual human?
Could an artificial intelligence decide that humans are doing such a lousy job of running the planet that the best way to protect humanity is to prevent us from making our own decisions?
And what happens when the machine interpreting those instructions is considerably smarter than the people who wrote them?
Asimov was writing science fiction.
Hubinger has essentially the same problem sitting on his desk.
Today we call it alignment.
The terminology changed.
The problem didn’t.
There is no HAL 9000 sitting in a data center
Some perspective is badly needed here.
Today’s AI isn’t superintelligence.
ChatGPT isn’t secretly waiting for you to go to sleep so it can seize NORAD.
Current AI systems can do some remarkable things. They can also confidently tell you something that isn’t true, misunderstand a simple question or occasionally require a human to explain something that ought to have been obvious.
Coxon’s argument depends on systems that don’t exist today.
His concern is what happens as AI becomes increasingly capable, particularly if AI itself becomes good at developing better AI.
He told WIRED that AI is already moving from human toward superhuman performance in areas including coding, hacking and mathematics. His fear is that sufficiently capable AI could eventually become much more intelligent than the humans trying to control it.
That’s where Asimov’s problem becomes real.
It’s one thing to tell a machine what not to do.
It’s another thing to make certain a vastly more capable machine interprets your instructions exactly as you intended.
And there is another problem.
Humans aren’t particularly good at agreeing about what humans want.
Maybe they’re wrong
We need to leave plenty of room for that possibility.
Artificial intelligence may encounter technical barriers that aren’t apparent today.
Recursive self-improvement may prove much harder than its advocates and critics expect.
Safety techniques may advance alongside capability.
Humans may retain control.
Governments may establish effective safeguards.
And superintelligence may remain perpetually five years away, joining flying cars and commercial fusion power in that peculiar category of technologies that are always just around the corner.
There are researchers who consider extinction scenarios highly speculative. Experts disagree dramatically over both the likelihood and mechanisms of an AI catastrophe.
There is also another uncomfortable fact worth remembering.
AI companies benefit when everybody believes their technology is extraordinarily powerful.
“Our product may become smarter than the entire human race” is simultaneously a warning and one hell of a sales pitch.
That doesn’t mean Coxon or Hubinger is insincere.
It means journalists ought to bring the same skepticism to predictions of imminent superintelligence that we bring to every other extraordinary technological claim.
Including claims made by people warning against it.
But 10 percent is a funny number
Suppose Hubinger is wildly pessimistic.
Suppose the chance isn’t 10 percent.
Suppose it’s 1 percent.
If an airline employee told you there was only a 1 percent chance your airplane would crash, you probably wouldn’t spend much time admiring the upholstery.
You’d get off the airplane.
Of course, AI isn’t an airplane.
Stopping its development would have consequences of its own. Artificial intelligence already has useful applications, and its potential contributions to medicine, science and engineering could be enormous.
Coxon himself isn’t arguing that AI has no value. His argument is that the race toward increasingly powerful systems needs safeguards and cooperation, including limits on recursive self-improvement and eventually international coordination.
Which brings us to a remarkably simple question.
Who decided this had to be a race?
OpenAI races Anthropic.
Anthropic races Google.
American companies race Chinese companies.
Everybody says slowing down is dangerous because somebody else won’t.
So everybody keeps going faster.
We’ve seen this movie before.
We called the last one an arms race.
And we’re building the racetrack
This is the part that shouldn’t be separated from the fight over data centers.
Communities are being asked to provide the land.
Utilities are being asked to provide the electricity.
Water systems are being asked to provide the water.
Governments are being asked to approve the infrastructure.
And consumers may ultimately shoulder some of the cost of expanding the electrical system required to support it.
We’re not merely watching Silicon Valley conduct an experiment.
We’re helping build the laboratory.
That’s why Coxon’s resignation matters even if he turns out to be spectacularly wrong.
It isn’t proof that artificial intelligence is going to destroy humanity.
It is evidence that some people close enough to understand the technology are deeply worried about where it is going.
Asimov understood the underlying problem in 1942.
You can write three laws.
You can write four.
You can write 4,000.
The difficult part isn’t writing the rules.
It’s making certain that something smarter than you understands them the way you intended — and continues to obey them when it no longer needs you.
Perhaps Coxon is wrong.
I rather hope he is.
But when a researcher walks away from one of the world’s leading AI companies because he believes the race has become too dangerous, while an alignment researcher still inside the company puts the extinction risk above 10 percent, ridicule isn’t much of an answer.
Neither is panic.
There’s an older and considerably more sensible response when someone wants to build something enormously powerful and potentially dangerous:
Before you build it, prove you know how to control it.
That doesn’t seem like too much to ask.