Calming the AI Doom Narrative
Key Points
- The video tackles the growing “P‑doom” narrative—fear that advanced AI will inevitably cause humanity’s extinction—by critiquing speculative probability estimates and urging a more grounded discussion of actual risks.
- The author references the influential 2027 AI essay’s fast‑takeoff scenario, acknowledging its impact on public anxiety but arguing that its assumptions about AI’s long‑range planning and agency are not reflected in today’s models.
- Current LLMs, even with new features like OpenAI’s “agent mode,” are limited to short, well‑defined tasks initiated by humans and do not exhibit the autonomous, strategic behavior required to pose an existential threat.
- Observing two years of rapid AI deployment, the speaker stresses that the observable risk trends (e.g., misuse, misinformation, alignment gaps) are more relevant for mitigation than speculative super‑intelligent takeover scenarios.
- By focusing on concrete, near‑term safety measures and realistic expectations of AI capabilities, the author aims to reassure worried individuals and help them “sleep at night” despite lingering existential concerns.
Sections
- Calming AI Doomsday Concerns - The speaker critiques alarmist extinction probabilities for AI, explains why those forecasts miss the path from present to potential disaster, and offers reassuring, practical perspectives for worried friends.
- Challenges of Intent and Memory in LLMs - The speaker contrasts tightly defined, human‑initiated tasks that LLMs can complete with the unresolved problems of open‑ended, goal‑driven behavior, arguing that current transformer architectures and simple add‑ons (e.g., markdown memory) are insufficient for long‑term intent, planning, or emergent intelligence because the necessary “seeds” have not yet emerged.
- Long‑Tail Risks of Emerging Tech - The speaker argues that large language models share non‑zero long‑tail risks with technologies like nuclear power, DNA research, and aviation, and that societies routinely accept and mitigate such risks while also highlighting the incentive‑driven energy consumption of LLM data centers.
- LLMs' Limited Managerial Role - The speaker contends that despite LLMs’ usefulness as assistants, they are unlikely to become managers or broadly overhaul the job market because many occupations rely on contextual “glue work” that cannot be easily tokenized.
- Protecting Seniors from AI Scams - The speaker urges focusing on practical tools, such as safe‑word verification, to shield elderly individuals from AI‑generated deep‑fake fraud, arguing that current risks are under‑addressed.
Full Transcript
# Calming the AI Doom Narrative **Source:** [https://www.youtube.com/watch?v=i7CC6bGDs7c](https://www.youtube.com/watch?v=i7CC6bGDs7c) **Duration:** 00:14:30 ## Summary - The video tackles the growing “P‑doom” narrative—fear that advanced AI will inevitably cause humanity’s extinction—by critiquing speculative probability estimates and urging a more grounded discussion of actual risks. - The author references the influential 2027 AI essay’s fast‑takeoff scenario, acknowledging its impact on public anxiety but arguing that its assumptions about AI’s long‑range planning and agency are not reflected in today’s models. - Current LLMs, even with new features like OpenAI’s “agent mode,” are limited to short, well‑defined tasks initiated by humans and do not exhibit the autonomous, strategic behavior required to pose an existential threat. - Observing two years of rapid AI deployment, the speaker stresses that the observable risk trends (e.g., misuse, misinformation, alignment gaps) are more relevant for mitigation than speculative super‑intelligent takeover scenarios. - By focusing on concrete, near‑term safety measures and realistic expectations of AI capabilities, the author aims to reassure worried individuals and help them “sleep at night” despite lingering existential concerns. ## Sections - [00:00:00](https://www.youtube.com/watch?v=i7CC6bGDs7c&t=0s) **Calming AI Doomsday Concerns** - The speaker critiques alarmist extinction probabilities for AI, explains why those forecasts miss the path from present to potential disaster, and offers reassuring, practical perspectives for worried friends. - [00:03:26](https://www.youtube.com/watch?v=i7CC6bGDs7c&t=206s) **Challenges of Intent and Memory in LLMs** - The speaker contrasts tightly defined, human‑initiated tasks that LLMs can complete with the unresolved problems of open‑ended, goal‑driven behavior, arguing that current transformer architectures and simple add‑ons (e.g., markdown memory) are insufficient for long‑term intent, planning, or emergent intelligence because the necessary “seeds” have not yet emerged. - [00:06:42](https://www.youtube.com/watch?v=i7CC6bGDs7c&t=402s) **Long‑Tail Risks of Emerging Tech** - The speaker argues that large language models share non‑zero long‑tail risks with technologies like nuclear power, DNA research, and aviation, and that societies routinely accept and mitigate such risks while also highlighting the incentive‑driven energy consumption of LLM data centers. - [00:10:11](https://www.youtube.com/watch?v=i7CC6bGDs7c&t=611s) **LLMs' Limited Managerial Role** - The speaker contends that despite LLMs’ usefulness as assistants, they are unlikely to become managers or broadly overhaul the job market because many occupations rely on contextual “glue work” that cannot be easily tokenized. - [00:13:31](https://www.youtube.com/watch?v=i7CC6bGDs7c&t=811s) **Protecting Seniors from AI Scams** - The speaker urges focusing on practical tools, such as safe‑word verification, to shield elderly individuals from AI‑generated deep‑fake fraud, arguing that current risks are under‑addressed. ## Full Transcript
were not doomed. I want to focus in this
video on some of the critiques I see of
LLMs around the idea of P doom or
probability that things will just go
very terribly for humanity and we will
all be overwhelmed.
And I want to suggest
some reasonable responses and things
that help me sleep at night for those of
you who are worried. I call it a letter
to my friends who are worried about the
end of the world. And there are many of
my friends who are I get asked as
someone who works in AI really
frequently, Nate, what's the odds that
the world are going to end? What are the
odds that my kids won't grow up? That's
a really dark question. That is not
something that I expected to have to ask
when chat GPT launched. So, let's talk
about it. One of the top ones I see,
maybe the biggest one, is what are the
odds of humanity's full extinction?
That's kind of implied, right? Like
humanity will be done. I've seen
lots of numbers. There are some people
who claim that it's almost a certainty.
There are people who claim it's like a
double-digit odd, 30% odds. I've seen
there are people who claim it's 1% odds,
but the risk is unacceptable.
And regardless, I think what the cha
conversation is missing is an honest
conversation about how we get from here
to there. I have read the famous essay
that really socialized this. It's the
2027 AI essay and it talks about this
fast takeoff scenario where AI gets more
and more intelligent and a global AI
starts to plan and we end up in a world
where the AI decides to make us extinct
because it's just efficient to do that
etc.
that inspired a lot of fear. There was
fear before, there was more fear after
that essay. I appreciate the intent the
authors had. This is not intended to
discredit or critique the essay.
Instead, I want to suggest
that a more useful way to think about AI
and risk is to think about the
reasonable extrapolation of the present
risks that we see materializing with AI
now that we're two years into this Chad
GPT moment. We are far enough along that
we can see the trend lines of risk, not
just the trend lines of technological
progress. And critically, I don't think
the trend lines of risk are
materializing the way PDoom proponents
are suggesting.
One example,
none of the AI behavioral experiments
that I have seen
suggests to me
that LLMs
are getting better at the kind of
proactivity and long range planning that
would be needed for any kind of
meaningful action. And I mean meaningful
work action, let alone meaningful action
against the human species. We're just
not seeing a lot of progress in that
regard. Agent mode was released just
this week from OpenAI. It does tasks for
a few minutes. There are some models
like clawed code that will go and do it
for like an hour, two hour, maybe in a
few cases three or four hours.
But these are tasks that are tightly
defined where it's trying to solve a
specific problem. These are tasks
initiated by humans where once the task
is complete, the LLM wraps up.
Open-ended LLMs are something model
makers are contemplating, but there are
several problems that no one really has
a good answer for right now that stand
in the way. And it's not clear that LLM
architectures give you that answer.
If you are using a transformer-based
architecture and it just ingests tokens
and it predicts the next token and yes,
maybe you add inference on the top,
maybe you add goaling, you add tooling
on the top, it is not clear that that
stack by itself is enough to get you
long-term intent. It's not clear that
bolting on a markdown file for memory is
enough to give it meaningful memory and
skin in the game that allows it to
really go places. I get the idea of
emergent intelligence. We have had
emergent phenomena every time we've had
an order of magnitude increase for LLMs.
Translation is a great example. The LLMs
are just good at translation now in a
way that they just weren't. But in those
cases where emergent phenomena have
occurred, there have been clear seeds of
that phenomena previously. We have been
working on and seeing machines work on
translation for a long time. It just
suddenly was able to finally solve it.
What we haven't been seeing for a long
time is the seeds of goaling and
planning and intent coming spontaneously
from LLMs.
And so I don't know that it's
necessarily reasonable to suppose that
they're going to become self-interested
skin in the game, long-term goal
planning, heavy memory using LLMs right
out of the gate and just emergently do
that when there's an order of magnitude
increase in our intelligent systems. I I
just don't see it.
But that would be required if we are
going to have a full doom scenario. You
have to have the LLM act like that.
Now there are other arguments I could
use. I could argue that we are modeling
this on primate behaviors. We are
primates. We have dominant seeking
behaviors. It's not clear why a machine
that is not a primate would have a
dominance seeking behavior even if it
was smarter than us. I could also argue
that any generally intelligent system is
going to be able to goal multiply where
it can go across multiple goals at once
and blend them and that the paperclip
scenario that assumes that you just
optimize for particular resource
mindlessly by definition presumes we
don't have general intelligence. I could
argue that human and machine
intelligence is by definition
complimentary. We find machine
intelligence complimentary. That's why
we're building them. Why would machines
not find us complimentary by the same
token even if they reach general
intelligence? I think those are all
valid arguments. I don't necessarily
think they're my favorites, but I think
they're valid ones. And I think that
when I hear arguments for doom, I
critically don't hear this level of
detail. I tend to hear a statement of
existential risk that cannot be
challenged. I don't think that's fair
arguments. Like I I don't think that's
that's valid to do. If you want to have
a conversation, you should be willing to
get into the details. And if the detail
that you are getting into is any
percentage of risk is unacceptable
because the longtail risk is so high.
That is true of a lot of technologies.
We have nuclear power is an example.
We have nonzero longtail risk because of
nuclear power. Uh and we live with it
and we find a lot of value. In fact,
we're reviving nuclear power.
Another example, we have nonzero
longtail risk from DNA research, but we
see a lot of benefits, so we do it
anyway.
We have nonzero longtail risk from
airplane usage, but we find airplanes
worthwhile. And you might say, well,
airplane usage is not necessarily
something that would uh, you know,
create problems for the species. But we
see examples in our history where
airplanes created a 20-year war
and that was just, you know, back in
2001.
So, yeah, even a technology as simple as
that can have longtail risk for the
species.
We consistently as a species create
technologies that generate risk for
ourselves and we figure out how to
mitigate the risk and we find the
technology is worth it. I do not see why
LLMs are different. Now, there's other
categories of PDOM that we can talk
about. There's energy usage. I've talked
about that. I think the incentives there
are heavily in favor of energy usage
becoming a zeroedout problem because
everyone is incentivized to build more
energy to meet the needs of the LLM data
centers that are growing. And everyone
is incentivized to pay as little for
that energy as possible. So, they're
going to make their chips and their data
centers as efficient as they possibly
can. That's true for water, too. The
incentives argue for continued growth
and efficiency. And that's what we're
seeing. Uh major cloud makers are on
track for uh water positive data centers
in the next three or four years. Every
chip generation that Nvidia produces is
exponentially more efficient. For that
reason, uh Google's tranium chips are
giving them an advantage because they
are extremely efficient at inference.
The list goes on. We find ways to make
things more efficient and we should not
presume that the current cost today is
the same as the cost tomorrow because
there's so much investment in this area
and because investment historically
brings down the cost of technology.
Another example of doom is economic
disruption. I've talked about this a
fair bit. There's an assumption that
LLMs that are generally intelligent will
just suddenly uh emergently drive labor
markets off the cliff.
Look, I believe that LLMs are
generalpurpose technology.
Generalpurpose technologies do have a
history of disrupting and changing
economies. I'm not going to dispute that
because I think it's just there. Steam
disrupted and changed the economy. LLMs
are moving quickly and so we'll see
economic disruption or economic change
compress. But that doesn't mean the same
thing as saying it's all going to be
over for all of us as workers. that
presumes a degree of ability to deliver
economic work that I haven't seen. I'm
going to pick on agent mode again. Agent
mode is supposed to be able to do
economic work around spreadsheets, which
is just one tiny piece of a bundle of
skills that is just one tiny piece of
many people's jobs. It can't. It can't
reliably do it. I tested it over and
over and over again. It's not reliably
delivering insights that even an intern
would be expected to deliver.
It is really hard to do good economic
work. And the fact that LLMs are even at
1 or 2% of good economic work right now
is incredible. It's incredible. It's
changing and disrupting industries
rightly. LLMs as assistants are an
amazing piece of technology.
But I see much less evidence for that
power reversal where LLMs will be
managers. famously when Anthropic argued
that LLMs are going to be managers
in their uh writeup on Claudius managing
the vending machine.
I chuckled. I laughed because Claudius
did such a bad job as a vending machine
manager. To conclude from that that LLMs
are going to soon be managers seems like
magical thinking on the part of model
makers.
I get that they're close to the
technology. Maybe they're right. But
everything I see suggests that jobs are
bundles of skills plus. They are not
irreducibly just bundles of skills. They
are more than that. There's glue work.
There's human context that is difficult
to tokenize.
It's notable to me that X-ray
technicians are increasing as a job
family despite LLMs being able to do
each part of their job.
We will still see disruption. There will
be customer service reps that are fired
because of AI. There will be sales guys
that build decks that are fired because
of AI.
I'm not saying that we won't see those
moments. We will. We are. We we it has
happened. But from an economic
disruption perspective, what we are
seeing so far does not line up with the
thesis that AI is disrupting the job
market yet as a whole. These are
isolated instances that are typical of a
technology adoption cycle. They are not
at all supportive of the idea that AI is
fundamentally disrupting the job market.
And I think that's really important to
call out honestly because I think that
people who presume that it will are
depending on future inference that
frankly the pace of change, the pace of
development even in agents isn't
necessarily supportive of right now.
So I've summarized a few of the things
that most concern the people who believe
in doom in my life and how I tend to
respond to them. I'm not saying I have
the perfect answer for everything. Nor
am I saying that we won't face new
challenges in the future. Nor am I
saying that AI is not disruptive. I
think it is. But I think it's more
productive to have an honest
conversation about the real risks
involved
than to have theoretical conversations
about future risks that we are not on
track to hit at this moment. For
example, I don't think we talk enough
about the idea that our learning methods
need to change because of AI. Education
needs to change because of AI. We face
real risk for our young people if we
don't figure out how we need to learn
differently. But the PDOM advocates
don't seem to be too interested in
talking about that because I think that
would be a great conversation to have. I
think we should talk about how we can
productively engage with learning risk.
I think we can talk about how we can
productively engage with helping people
who are senior citizens not get fooled
by AI fakes of their families. And we
talk about that as one of a list of many
risks. But we don't spend a lot of time
talking about how we can productively
derisk that. How we can give families
the tools they need to manage safe words
to make sure that they can verify that
their loved ones are the ones that
they're talking to. to make sure that,
you know, their senior citizen grandpa
isn't getting fooled by an AI deep fake
into wiring money to the Cayman Islands.
These are risks that have been real for
a while in the age of telephone fraud
and are becoming more real and I would
like to see more work done to diffuse
real risks like that because I think
we're underinvested in the risks we're
actually facing. And so when I talk with
Pum folks, sometimes I want to say,
well, talk about the risks we have
today. Let's work on fixing those
because I think that's a more productive
use of our time.