There is a scene in the movie version of Carl Sagan’s book Contact where
the main character, an astronomer who has detected the first radio
signal from an alien civilization, is being considered for the role of
humanity’s representative to meet the aliens. The international panel
interviewing her asks, “If you could ask [the aliens] just one question,
what would it be?” Her reply is: “I’d ask them, ‘How did you do it? How
did you evolve, how did you survive this technological adolescence
without destroying yourself?” When I think about where humanity is now
with AI—about what we’re on the cusp of—my mind keeps going back to that
scene, because the question is so apt for our current situation, and I
wish we had the aliens’ answer to guide us. I believe we are entering a
rite of passage, both turbulent and inevitable, which will test who we
are as a species. Humanity is about to be handed almost unimaginable
power, and it is deeply unclear whether our social, political, and
technological systems possess the maturity to wield it.
In my essay Machines of Loving Grace,
I tried to lay out the dream of a civilization that had made it through
to adulthood, where the risks had been addressed and powerful AI was
applied with skill and compassion to raise the quality of life for
everyone. I suggested that AI could contribute to enormous advances in
biology, neuroscience, economic development, global peace, and work and
meaning. I felt it was important to give people something inspiring to
fight for, a task at which both AI accelerationists and AI safety
advocates seemed—oddly—to have failed. But in this current essay, I want
to confront the rite of passage itself: to map out the risks that we
are about to face and try to begin making a battle plan to defeat them. I
believe deeply in our ability to prevail, in humanity’s spirit and its
nobility, but we must face the situation squarely and without illusions.
As
with talking about the benefits, I think it is important to discuss
risks in a careful and well-considered manner. In particular, I think it
is critical to:
- Avoid doomerism. Here, I
mean “doomerism” not just in the sense of believing doom is inevitable
(which is both a false and self-fulfilling belief), but more generally,
thinking about AI risks in a quasi-religious way.1 Many
people have been thinking in an analytic and sober way about AI risks
for many years, but it’s my impression that during the peak of worries
about AI risk in 2023–2024, some of the least sensible voices rose to
the top, often through sensationalistic social media accounts. These
voices used off-putting language reminiscent of religion or science
fiction, and called for extreme actions without having the evidence that
would justify them. It was clear even then that a backlash was
inevitable, and that the issue would become culturally polarized and
therefore gridlocked.2 As of 2025–2026, the pendulum has
swung, and AI opportunity, not AI risk, is driving many political
decisions. This vacillation is unfortunate, as the technology itself
doesn’t care about what is fashionable, and we are considerably closer
to real danger in 2026 than we were in 2023. The lesson is that we need
to discuss and address risks in a realistic, pragmatic manner: sober,
fact-based, and well equipped to survive changing tides.
- Acknowledge uncertainty. There
are plenty of ways in which the concerns I’m raising in this piece
could be moot. Nothing here is intended to communicate certainty or even
likelihood. Most obviously, AI may simply not advance anywhere near as
fast as I imagine.3 Or, even if it does advance quickly, some
or all of the risks discussed here may not materialize (which would be
great), or there may be other risks I haven’t considered. No one can
predict the future with complete confidence—but we have to do the best
we can to plan anyway.
- Intervene as surgically as possible. Addressing
the risks of AI will require a mix of voluntary actions taken by
companies (and private third-party actors) and actions taken by
governments that bind everyone. The voluntary actions—both taking them
and encouraging other companies to follow suit—are a no-brainer for me. I
firmly believe that government actions will also be required to some extent,
but these interventions are different in character because they can
potentially destroy economic value or coerce unwilling actors who are
skeptical of these risks (and there is some chance they are right!).
It’s also common for regulations to backfire or worsen the problem they
are intended to solve (and this is even more true for rapidly changing
technologies). It’s thus very important for regulations to be judicious:
they should seek to avoid collateral damage, be as simple as possible,
and impose the least burden necessary to get the job done.4
It is easy to say, “No action is too extreme when the fate of humanity
is at stake!,” but in practice this attitude simply leads to backlash.
To be clear, I think there’s a decent chance we eventually reach a point
where much more significant action is warranted, but that will depend
on stronger evidence of imminent, concrete danger than we have today, as
well as enough specificity about the danger to formulate rules that
have a chance of addressing it. The most constructive thing we can do
today is advocate for limited rules while we learn whether or not there
is evidence to support stronger ones.5
