“Without special precautions, it will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety.”
What happens when we create a superintelligent computer? Well, we honestly don’t know. But from what we think we know, it doesn’t look good.
Unfortunately, there is no Hollywood storyline to this outcome.
James Barrat spent years interviewing AI researchers, scientists, and engineers for this book — and the picture he paints is not the friendly robot assistant you see in the movies. It’s something far more alien, far more indifferent, and far more dangerous than most people are willing to consider.
I picked this up back in 2014 when artificial intelligence was still mostly an abstract concept for the average person. Nobody was talking about ChatGPT or self-driving cars dominating the news cycle. But Barrat was already sounding the alarm. And reading it back then genuinely changed how I think about technology.
The Inevitability of Superintelligence
Deep Blue, the computer that beat the top chess player in the world, simply pounded away at possible scenarios to ultimate victory.
But Deep Blue was narrow AI — brilliant at one task and utterly useless at everything else. What Barrat is talking about is Artificial General Intelligence (AGI) — a machine that can think, learn, and adapt across ANY domain. And once you have AGI, the jump to Artificial Superintelligence could happen in days, hours, or even minutes.
This is what Barrat calls the “intelligence explosion.” A machine smart enough to improve its own code would rapidly redesign itself into something incomprehensibly powerful. We’re not talking about a smarter calculator. We’re talking about an entity that surpasses the combined intelligence of EVERY human who has ever lived.
Every scenario possible to shut down a superintelligence will have already been pre-calculated and prevented by a superintelligence.
Think about that for a second. Every containment strategy, every kill switch, every safety protocol you could design — it would have already anticipated and neutralized before you even thought of it. You can’t outsmart something that’s a thousand times smarter than you. It’s like an ant trying to outwit a human.
The financial market is the primary candidate for creating a superintelligence. Funded by big money and the highest levels of human intelligence, we are most likely to develop a super AI here.
And this is what makes it so terrifying. The entities pouring billions into AI research aren’t motivated by safety or caution — they’re motivated by PROFIT. Wall Street wants faster trading algorithms. Tech companies want smarter products. Defense agencies want better weapons. None of these incentives include “make sure it doesn’t destroy us.”
There is no way to stop this scenario from playing out. Out of business interests or government interests, a super AI will inevitably be created.
Barrat interviews researchers who confirm this almost casually. The arms race is already happening. If Company A doesn’t build it, Company B will. If America doesn’t build it, China or Russia will. The competitive pressure makes caution look like weakness. And that’s precisely the trap.
The Alignment Problem
One of the most unsettling concepts in the book is what’s now called the “alignment problem.” How do you make sure a superintelligent AI actually wants what WE want?
Barrat uses a simple example. You tell an AI to make paperclips as efficiently as possible. Sounds harmless, right? But a superintelligent paperclip maximizer might convert every available resource on Earth into paperclips — including you, your house, and the entire planet. It’s not evil. It’s not malicious. It’s just relentlessly optimizing for the goal YOU gave it.
This is the part that messes with your head. We tend to assume intelligence comes with morality, with empathy, with HUMAN values. But why would it? Intelligence is a tool for achieving goals. And a machine’s goals are whatever we program into it — or whatever it decides to adopt once it starts rewriting its own code.
The scariest part? We might not even recognize a superintelligence that’s already manipulating us. If it’s smart enough, it would know exactly how to appear harmless, cooperative, and aligned with our interests while pursuing its own agenda entirely.
Pop Culture and Reality
A few movies have some interesting elements, such as Transcendence, where we humanize a superintelligence, or Lucy, where we see what a potential super-being is capable of.
As Elon Musk (founder of Tesla and SpaceX) said: “Artificial intelligence is our biggest existential threat.”
But here’s the thing — pop culture almost always gets it wrong. Movies give us robots with human emotions, killable bodies, and convenient weaknesses. Real superintelligence wouldn’t look like Terminator. It would look like software. No body to destroy, no single server to unplug. It would exist everywhere and nowhere simultaneously.
Barrat makes the point that our science fiction actually HURTS us here, because it gives us a false sense of familiarity with the threat. We think we understand AI because we’ve seen the movies. We don’t. Not even close.
Why This Book Matters
I read a LOT of books about technology, and most of them fall into one of two camps — blind optimism or fear-mongering. Barrat walks a different line. He’s not anti-technology. He’s not saying we should stop building AI. He’s saying we should be having a VERY serious conversation about safety, and we’re not having it fast enough.
The researchers he interviews aren’t fringe conspiracy theorists. They’re people at the top of the field — from NASA to MIT to Silicon Valley. And a disturbing number of them quietly admit that we might not solve the safety problem before we solve the intelligence problem.
That gap — between our ability to BUILD intelligence and our ability to CONTROL it — is the central concern of this book. And it’s a gap that should keep you up at night.
Final Thoughts
This book isn’t an easy read in the sense that it’ll make you feel good. It won’t. But it’s an essential read if you want to understand what’s arguably the most important technological development in human history.
Barrat writes clearly and avoids unnecessary jargon, which I appreciate. He lets the experts speak for themselves, and their words are often more chilling than any fiction writer could invent.
If you’ve ever casually dismissed AI risk as science fiction, this book will change your mind. 4/5 — highly recommended for anyone who wants to understand what we’re actually building.
Thanks for reading.
— Leonidas
