DeepSeek R1: Hype, Costs, Impact
Key Points
- The panel gave wildly different importance scores for DeepSeek R1 (5, 9, and 7.5), underscoring how contentious its impact currently is.
- DeepSeek R1, a new open‑source model from a Chinese lab, is being hailed as competitive with leading proprietary systems from Anthropic, OpenAI, etc., and has generated massive buzz—even reaching the hosts’ families.
- The show’s hosts set out to debunk common myths, noting that mainstream coverage often misstates key facts about the model and its significance.
- One widely cited claim is that state‑of‑the‑art models can now be trained for about $5.5 million; Kate confirms the figure appears in the paper but warns it is heavily caveated and not a simple “price tag.”
- Overall, the episode frames DeepSeek R1 as a hot topic that warrants careful analysis beyond hype, with open‑source ambitions and cost narratives that need nuanced interpretation.
Sections
- DeepSeek R1 Rating Debate - Panelists humorously rate the significance of DeepSeek R1, giving scores from 5 to 9, as they launch an episode of the Mixture of Experts show focused on the AI model.
- Training Cost vs Inference Efficiency - The speaker uses a marathon analogy to show that an impressive efficiency metric reflects only the inference runtime of a model, not the extensive training effort required, warning against misinterpreting it as a low barrier for building new models.
- Chain‑of‑Thought RL on a Laptop - The speaker explains that a brief fine‑tuning phase with high‑quality chain‑of‑thought data followed by reinforcement learning can dramatically boost model performance using only modest hardware, challenging the need for massive pre‑training and fueling discussion of reduced compute demand and its impact on NVIDIA.
- Rethinking Model Scaling & Pre‑training Costs - The speaker argues that although scaling laws lessen GPU demands, the ever‑growing size of foundational models like DeepSeek‑V3 still entails massive pre‑training expenses, emphasizing the need to separate the modest cost of a single training step from the overall billions‑dollar investment.
- Hype vs Experimentation in Model Development - The speaker argues that the AI hype culture prizes flashy results over the extensive experimental work required to train models, before transitioning to topics like distillation and the resurgence of reinforcement learning.
- Hybrid Distillation and RL for Small Models - The speaker discusses how pure reinforcement‑learning fine‑tuning of a 32‑billion‑parameter model fails to yield strong reasoning, requiring a hybrid pipeline of synthetic data distillation, chain‑of‑thought fine‑tuning, and RL to improve small‑model performance.
- Math Compiler Enhances Chain‑of‑Thought - The speaker explains how converting automatically generated equations into an abstract syntax tree, walking the tree to produce step‑by‑step explanations, and fine‑tuning a language model with these outputs yields exceptionally precise mathematical reasoning.
- Model Distillation and Architecture Translation - The speaker explains how knowledge from a large teacher model—potentially a Mixture‑of‑Experts—can be transferred to a smaller student model through distillation, highlighting its role in model compression and the shift from sparse to dense neural architectures.
- Distillation: The Open-Source Secret - The speakers explain how model distillation, a long‑standing technique now accessible through DeepSeek’s publicly licensed model, makes it effectively impossible to stop others from replicating large AI models despite the high associated costs.
- DeepSeek's Efficient AI and Model Distillation - The speaker observes that a reasoning‑boosted model may lose general performance, but praises DeepSeek’s push for cost‑effective, task‑specific distilled models using open‑source teacher models, while noting the challenge of preventing model copying, akin to biometric security.
- OpenAI's Steady Response to DeepSeek - The speaker reviews DeepSeek's launch, cites Sam Altman's tweet affirming OpenAI’s unchanged research roadmap, and debates whether the news will alter OpenAI’s strategic approach.
- Choosing and Trusting LLMs Amid Competing Strategies - The speakers compare OpenAI’s market dominance with DeepSeek’s hardware‑constrained, efficiency‑driven approach, debating how these differing strategies affect model selection, trust, and economics for enterprises.
- Constraints Drive Open‑Source Innovation - The speakers argue that limited hardware resources force researchers to devise creative techniques, and sharing these open‑source solutions accelerates advancement across the entire AI community.
Full Transcript
# DeepSeek R1: Hype, Costs, Impact **Source:** [https://www.youtube.com/watch?v=jC0MGFDawWg](https://www.youtube.com/watch?v=jC0MGFDawWg) **Duration:** 00:39:10 ## Summary - The panel gave wildly different importance scores for DeepSeek R1 (5, 9, and 7.5), underscoring how contentious its impact currently is. - DeepSeek R1, a new open‑source model from a Chinese lab, is being hailed as competitive with leading proprietary systems from Anthropic, OpenAI, etc., and has generated massive buzz—even reaching the hosts’ families. - The show’s hosts set out to debunk common myths, noting that mainstream coverage often misstates key facts about the model and its significance. - One widely cited claim is that state‑of‑the‑art models can now be trained for about $5.5 million; Kate confirms the figure appears in the paper but warns it is heavily caveated and not a simple “price tag.” - Overall, the episode frames DeepSeek R1 as a hot topic that warrants careful analysis beyond hype, with open‑source ambitions and cost narratives that need nuanced interpretation. ## Sections - [00:00:00](https://www.youtube.com/watch?v=jC0MGFDawWg&t=0s) **DeepSeek R1 Rating Debate** - Panelists humorously rate the significance of DeepSeek R1, giving scores from 5 to 9, as they launch an episode of the Mixture of Experts show focused on the AI model. - [00:03:07](https://www.youtube.com/watch?v=jC0MGFDawWg&t=187s) **Training Cost vs Inference Efficiency** - The speaker uses a marathon analogy to show that an impressive efficiency metric reflects only the inference runtime of a model, not the extensive training effort required, warning against misinterpreting it as a low barrier for building new models. - [00:06:12](https://www.youtube.com/watch?v=jC0MGFDawWg&t=372s) **Chain‑of‑Thought RL on a Laptop** - The speaker explains that a brief fine‑tuning phase with high‑quality chain‑of‑thought data followed by reinforcement learning can dramatically boost model performance using only modest hardware, challenging the need for massive pre‑training and fueling discussion of reduced compute demand and its impact on NVIDIA. - [00:09:21](https://www.youtube.com/watch?v=jC0MGFDawWg&t=561s) **Rethinking Model Scaling & Pre‑training Costs** - The speaker argues that although scaling laws lessen GPU demands, the ever‑growing size of foundational models like DeepSeek‑V3 still entails massive pre‑training expenses, emphasizing the need to separate the modest cost of a single training step from the overall billions‑dollar investment. - [00:12:23](https://www.youtube.com/watch?v=jC0MGFDawWg&t=743s) **Hype vs Experimentation in Model Development** - The speaker argues that the AI hype culture prizes flashy results over the extensive experimental work required to train models, before transitioning to topics like distillation and the resurgence of reinforcement learning. - [00:15:29](https://www.youtube.com/watch?v=jC0MGFDawWg&t=929s) **Hybrid Distillation and RL for Small Models** - The speaker discusses how pure reinforcement‑learning fine‑tuning of a 32‑billion‑parameter model fails to yield strong reasoning, requiring a hybrid pipeline of synthetic data distillation, chain‑of‑thought fine‑tuning, and RL to improve small‑model performance. - [00:18:33](https://www.youtube.com/watch?v=jC0MGFDawWg&t=1113s) **Math Compiler Enhances Chain‑of‑Thought** - The speaker explains how converting automatically generated equations into an abstract syntax tree, walking the tree to produce step‑by‑step explanations, and fine‑tuning a language model with these outputs yields exceptionally precise mathematical reasoning. - [00:21:35](https://www.youtube.com/watch?v=jC0MGFDawWg&t=1295s) **Model Distillation and Architecture Translation** - The speaker explains how knowledge from a large teacher model—potentially a Mixture‑of‑Experts—can be transferred to a smaller student model through distillation, highlighting its role in model compression and the shift from sparse to dense neural architectures. - [00:24:40](https://www.youtube.com/watch?v=jC0MGFDawWg&t=1480s) **Distillation: The Open-Source Secret** - The speakers explain how model distillation, a long‑standing technique now accessible through DeepSeek’s publicly licensed model, makes it effectively impossible to stop others from replicating large AI models despite the high associated costs. - [00:27:46](https://www.youtube.com/watch?v=jC0MGFDawWg&t=1666s) **DeepSeek's Efficient AI and Model Distillation** - The speaker observes that a reasoning‑boosted model may lose general performance, but praises DeepSeek’s push for cost‑effective, task‑specific distilled models using open‑source teacher models, while noting the challenge of preventing model copying, akin to biometric security. - [00:31:12](https://www.youtube.com/watch?v=jC0MGFDawWg&t=1872s) **OpenAI's Steady Response to DeepSeek** - The speaker reviews DeepSeek's launch, cites Sam Altman's tweet affirming OpenAI’s unchanged research roadmap, and debates whether the news will alter OpenAI’s strategic approach. - [00:34:38](https://www.youtube.com/watch?v=jC0MGFDawWg&t=2078s) **Choosing and Trusting LLMs Amid Competing Strategies** - The speakers compare OpenAI’s market dominance with DeepSeek’s hardware‑constrained, efficiency‑driven approach, debating how these differing strategies affect model selection, trust, and economics for enterprises. - [00:37:39](https://www.youtube.com/watch?v=jC0MGFDawWg&t=2259s) **Constraints Drive Open‑Source Innovation** - The speakers argue that limited hardware resources force researchers to devise creative techniques, and sharing these open‑source solutions accelerates advancement across the entire AI community. ## Full Transcript
On a scale from 0 to 10, how big of a deal is DeepSeek R1?
Kate Soule is Director of Technical Product Management for Granite.
Kate, welcome to the show.
What do you think?
I'm going to take maybe a little bit of a controversial position.
I'm going to say 5.
Chris Hay, Distinguished Engineer and CTO of Customer Transformation.
Chris, welcome back as always.
0 to 10.
What do you think?
9. 11 or 9.
9, but I'm not sure which is the bigger number.
Wow, that is a niche reference.
And finally, Aaron Baughman is IBM Fellow and Master Inventor, zero to 10.
Aaron, what do you think?
Yeah, that's a great question, and I think we're gonna be right
in between the other two at a 7.5.
All that and more on today's Mixture of Experts.
I'm Tim Hwang, and welcome to Mixture of Experts.
This is episode 40 of the Mixture of Experts show.
I'm really excited to meet this milestone with an all-star cast.
Each week, MoE is the place to tune into to hear the news and analysis
on the biggest headlines and trends in artificial intelligence.
And today we're all going to talk about DeepSeek-R1.
It's basically anything that anyone is talking about right now is DeepSeek-R1.
It's the talk of the AI chatter class, it's rocking markets, and
even my dad is texting me about it.
Um, so what I want to do is start the first segment with a little
bit of DeepSeek-R1 myth busting.
Um, if you've been anywhere around our, uh, AI, uh, in the last
week, you know the basic story.
Um, there is a, a Chinese lab, uh, DeepSeek that has released a new
model called R1, uh, that is both open source and competitive with the
state of the art models coming out of Anthropic, OpenAI and all the names
that we're really familiar with.
And there has been so much hype about this story that as I said,
even my dad's texting me about it.
And a lot of the mainstream coverage has actually been
getting a lot of the facts wrong.
So kind of what I want to start with is just to knock down a bunch of myths, so
we can kind of calibrate as we really kind of peel back and talk about this story.
And, Kate, I want to start with you, because I know you were, uh, angry
about this in the show Slack, and so I wanted to give you a chance to
kind of like, you know, let loose.
Um, I think the first meme that we've heard in a lot of this mainstream
news coverage is we can now train state of the art models for 5.5 million dollars.
And that's so crazy expensive relative to the kind of numbers
that we've heard before, right?
I think the Stargate price was a hundred billion dollars
or something crazy like that.
Um, so Kate, how true is that number?
Can we, can we really train models for 5.5 million dollars now?
So, first, the number is true, it's published in the, uh, presumably, it's
published in the paper, DeepSeek isn't necessarily hiding anything about this
number, it's heavily caveated if you, if you look at it, but the takeaway that
people are driving from this number is a little bit crazy, so, yes, training one
iteration of a base model, DeepSeek-V3, by the way, this all came out in December.
This isn't late breaking news as of last week.
Back in December, they trained this model.
They said one iteration of training it would cost about 5.6 million dollars.
But that's like saying if a startup could go and train a model for the same cost.
That's like saying if I'm going to go run a marathon that the only
distance I'll ever run is 26 miles.
The reality is you're going to train for months.
Practicing, training, running hundreds and thousands of miles potentially leading
up to that one race that then takes 26.2 miles.
And if you look at what that number is and take that metaphor even a step
further, it's like saying, okay, what if I'm running a race, but I take a
break every mile, I stop, I, you know, take a drink of water, I take a nap, I
come back the next day, I keep running.
And you only add up your time from the actual miles that you're running
in the race, not all your breaks.
That's like the equivalent of what this number represents.
It's a really valuable number to understand and impressive.
The parts that they're measuring, they did bring down a lot in efficiency.
But that number does not represent the cost to go and now train a model.
It's not like we're going to have startups now flooding, you know, the ecosystem
with their own version of 600 billion parameter mixture of expert models.
That's super helpful.
Yeah, and I think it's a great calibration.
I want to kind of pick up on that last thing that you said, which
is there is a lot that's new here from an efficiency standpoint.
And maybe Chris, I'll toss it to you for sort of the next meme that we're hearing
kind of flow around in this space, which is DeepSeek-R1 is a huge breakthrough.
Models are running way more efficiently than they used to.
You know, dot, dot, dot, DeepSeek is so far ahead.
I know you said that you, uh, felt like this model was a big deal, like
9.11, um, a bunch of a big deal.
Uh, but can you tell us a little bit about, like, has DeepSeek really unlocked
some novel things, and if so, how big of a deal are these novel things that they're
really uncovering with this new model?
I said 9.11 or 9.9, so clearly, Tim, you think 9.11 is the
bigger number out of those two.
Sorry, there's some uncertainty bars there.
I actually think it is a big deal, so I think there is a few things, which
is, we're sort of joining everything together there, so we're actually
saying, okay, here's the base model, and then there's the RL training
for the R1 part of that, right?
And actually, if we separate out the kind of the DeepSeek-V3 version from
the RL training there for a second, I think there is a big deal there.
Because the reality is... Nevermind the 5.5 million bucks, right?
You are going to be able to take an existing base model that has
been pre-trained and then you are going to be able to do RL
training over the top of that.
You're going to be able to take your cold start fine tune data.
So you can take a relatively small amount of data set and then put that on
top and train it to do amazing tasks.
And I know that myself, right?
Because I took like a tiny model myself, one and a half billion
parameters, absolutely tiny Qwen model.
And then I put maybe a thousand lines of SFT data, right?
And I got that thing to be able to tune math, right?
Basic arithmetic at the kind of same level as GPT 4o, right?
Just myself and I, I'm telling you right now, I love IBM.
They do not pay me five and a half million dollars.
That was on my laptop.
So, so this is a big deal and it's a big deal because the thing that they're
showing there is long chain of thought has a huge impact and accurate data because
actually even to the point of RL, they they started with pure RL training, right?
So they actually just said, here's your rewards for not doing anything else.
And, and we're going to train the model that way.
And then they went to say, actually, if we, if we do one round of fine tuning
with a really good chain of thought, a set of data, maybe a few thousand rows worth
of data, and then do RL training after that, then we get much better results.
So actually what they're showing is that we can maybe stop obsessing about
pre-training so much and we can get into this kind of post training world
and Inference time compute world and for that you don't even need five and
a half million dollars. Just your laptop and a little bit of tenacity and a
little bit of GPU is gonna do that job.
That's awesome yeah, we're gonna talk a little bit more about RL and chain
of thought in just a second, but Aaron I think before we move to that
one other question I want to ask you the kind of other third big meme is
that everybody suddenly discovered Jevons paradox, uh, this week.
Um, and I think one of the narratives that popped up is NVIDIA is doomed.
You need a lot less compute, uh, compute for these models now.
You know, NVIDIA's stock price took a tumble.
I bought the dip for what it's worth.
Um, and I want to say to, uh, I guess for Aaron, if you wanted to kind of
respond to this, this question, or whether or not you think it's a myth
at all, is, um, are we going to need a lot less compute in the future?
Is NVIDIA doomed?
Like how should we read this?
And if you want to explain Jevons paradox, you can go ahead too.
Yeah, I mean, I mean, so, so I think, you know, fundamentally, you know,
that that's an interesting notion.
Um, you know, but I tend to follow, um, the dynamics of AI, you know, which
comes in to me three different areas.
One is the scaling law, right?
Is that, you know, it tends to say that, um, as you scale up the train
of AI systems, you get better results, which means bigger models are better.
Generally, right?
And then the shifting curve where new ideas are making
training more efficient, right?
And so this, you know, affects that scaling law.
So the more new ideas you get, you know, the smaller models
become more powerful, right?
But then there's a third, which is a shifting paradigm of these big
revolutionary ideas and can an order of magnitude change the scale of which
you actually need to train these models in order to get performance, right?
And so I think by having those points one, two, and three laid out, uh, which
is, you know, backed by a lot of research over time, you know, you know, I think
that, yes, there, there's always going to be a demand for GPUs, but I do think
that, um, there's going to be different chip architectures that are coming out,
but also, um, if you look at some of the efficiency gains that- that V2 had, um,
such as a multi-head attention, were they able to cache a lot- a lot of the weights?
The, the token throughput is incredible, uh, that they were able to achieve,
but I think that was one of the bigger innovations that they had.
Uh, and then the second one was there, what they call the DeepSeek, um, MoE, uh,
where they're able to sort of partition out and share uh, knowledge amongst these
different agents that they can have.
And that also helps, but those two things where some of those, um, uh,
pieces that gave us the shifting curve on that scale law, which said,
okay, I don't need as many GPUs now.
But if you look at the foundational model, um, so if you go to,
let's say, DeepSeek-V2, right?
It's big.
It's a very big model and v3 is even bigger with a- what is it?
671 billion parameters, right?
That's a very big model.
Yeah, it's, it's chunky.
Yeah.
So, I mean, I mean, that that's, it's very fun, right?
To watch that curve.
And I, and I think that we'll see agglomeration of models together, we
can do reverse distillation, right?
To create and combine smaller models together.
Uh, you can do model distillation to create smaller models, you know, um,
but, but it's, it's going to be fun.
I want to maybe pick out a point that Chris actually mentioned that
I think is really important, which is the like, can we just stop
worrying about pre-training now?
Because I think everyone is talking about this 5.5, 5.6 million dollar number.
And they're tagging it to all of these amazing performance improvements
that we're seeing in the R1 model and that, and the distilled models, and
they're kind of equating the two and saying, all right, now we can just
go and we are getting like this crazy performance at a pretty minimal cost.
And I think it's really important to disambiguate these two things, right?
A step in this process costs about 5.6 million, the true cost of building this pre-training
model is likely orders of magnitude higher, but Regardless, it almost
doesn't matter, like this 5.6 million number doesn't even matter because you
can take this big model that's now open source and distill it basically for free
on top of other open source, smaller models that are out there to get crazy
performance improvements and build.
So it's not that startups are going to go and build and pre-train their own
600 billion parameter model because the cost is only 5.6 million dollars.
That's the wrong takeaway.
It's that we now have the ability to distill and thanks to more and more
competitive models being put into the open source that distillation is
becoming even more powerful and use reinforcement learning as a technique
that DeepSeek used really effectively to go and build our own smaller versions of
these models that are really powerful.
And that is where there's actually very low barrier to entry now, as Chris is
saying, you know, doing it on your laptop.
Yeah, yeah, yeah.
I find that, you know, very, very nice because it's like a house of cards, right?
I mean, they're only quoting the top card.
They're not quoting any of the cards at the bottom.
And if you move one of those cards at the bottom, the whole house collapse, right?
And so that 5.5 million is only the cost associated with maybe you know,
you know, one, one epoch right of this type of training and that's it.
But if you look at the hardware, even that they use, what are the H800's?
Just procuring those alone or using those as a service is expensive, right?
Um, and so, so they're excluding lots of costs associated with
prior research ablation studies and lots of different things, right?
Which um that that number is very, very much misleading.
Uh, Chris, I see you nodding.
Do you want to jump in?
I don't know if you have a comment
Uh, no, I was just nodding in agreement because I'm a very
kind and collaborative person.
Uh, no, I, I, I, no, I, I, I absolutely agree.
I, I think that, um, you're gonna go for the big hit numbers, right?
You're gonna say, we did this super cheap.
And you are really going to miss out all the steps that took you to
get there in the first place, right?
And, um, and as Kate probably knows better than anyone, right, that the
amount of experimentation that it takes for these models, right, to
get to the final version is a lot.
So the, the actual final epoch, as Aaron was saying, that final training
run, that's just, that's just the kind of the end of the road there.
So, um, but.
You know what?
No one wants to hear about the big journey going up there.
They want to hear the big number.
We're in a hype industry, baby.
So we'll, yeah, five and a half million.
Here we go, right?
Kate, I guess maybe one last myth I've seen kind of popping up that might
be good to address before we, we do a segment on distillation, because it's
already come up a couple times, and I think it is worthwhile to explain
why, what it is and why it matters.
But maybe one last thing to cover before we get to that, Kate, is, um, on the
point about RL, um, it feels like, the DeepSeek narrative has also been a little
bit about like the revenge of RL, like reinforcement learning's back, baby.
Um, and I know some people have gone so far to be like, everything is RL now.
You know, fine tuning is dead.
Um, do you want to talk a little bit about that?
Like even, you know, with everything that we've said, like how much does R1 indicate
to us that really RL will be kind of like the more dominant method for these types
of fine tuning efforts going forward?
Yeah, and I'm really curious to get Chris's take on this because I know he's
just run these experiments right locally on his own laptop, but so DeepSeek in
their paper, they trained two models, uh, in addition to all the smaller distilled
models that they, that they worked on.
One model was trained with just reinforcement learning only.
So there's no additional data that's added.
You've got your pre-trained model, which costs, you know,
5.6 million plus, you know, all the arguable buffer on top.
And they just use reinforcement learning using some rules based systems more or
less to be able to verify the results, um, and, and score the responses.
And then, and so they called that R1-Zero, I believe.
Then there's R1, which they also created because in their paper they
mentioned that there were some, you know, rough edges, so to speak, on the,
the reinforcement learning-only model.
And in that model, they start the model first with some fine tuning,
basically using some structured data in order to better prime it for
this reinforcement learning task.
And that is the model that everyone's now playing with on the DeepSeek app and
that everyone's really excited about.
So I think it's a really interesting look at, you know, the takeaway
shouldn't be that, oh, we, you know, can't do RL only we had to, you know,
resort to this cold start and fine tuning before the model was released.
The takeaway that I think people should have is it's amazing how
far they were able to push just RL.
And yeah, there's still always going to be a need for some
structured data potentially.
And there's, you know, maybe a hybrid approach is best, but it is kind of
crazy how far they were able to push it.
Now, what they also published in their paper, getting to the distilled models,
and you asked about distillation, distillation has been around forever.
It's where, you know, back to, you know, early days of the first Llama model,
you know, a group of students distilled that into Vicuna, um, and it's basically
where you generate a bunch of synthetic structure data from a big model and
use that to fine tune a small model.
So DeepSeek used that same kind of thought process doing just RL only on a small
model, so no big models involved, just RL and try to see how far could they get, you
know, they published numbers on Qwen32B.
So how far can we push Qwen32B's reasoning just on RL?
And they weren't able to, in the paper, they claimed they weren't able to
push it nearly as far, get any real reasoning capabilities out of the model,
they had to resort to distillation, take their big R1 model, generate a
bunch of synthetic data, and tune it.
So, you know, I'm curious from your perspective, like, what your take
is on that, Chris, based off of some of the RL experiments you've
been doing with small models.
You said you also, I think, did some fine tuning first to start it
off and then with chain of thought reasoning and then RL on top.
Now, for me, the critical thing is the long chain of thought reasoning.
That is actually an accurate long chain of thought reasoning.
That is the thing that really enabled everything.
So, again, if you look at the paper, when they did RL, um, they said they got there.
But, you know, if you think about, especially math problems, LLMs
are not really good at that.
So you're going to say, what's 25 plus 8?
What's this?
Whatever.
And you're going to ask an LLM, you know, to go and generate me this sum.
And it may or may not get it right.
It may or may not get the sums and the length of chain of thoughts that you want.
It may not get its explanations right.
So it's, it's really a, a bit of a crap shoot and getting
an accurate chain of thought.
And then at the end of it, they're using this thing called a verifier.
And what the verifier does is, is, you know, take the answer that you've got
and go you know, run a bit of rules to run the equation and say, yeah,
that was correct or that was wrong.
And then you get a, you get a bit of a reward, you know,
it's like, here's a cookie.
Well done model.
Good job.
But, but if you think about how long that's going to take it,
it, you really are monkeys and typewriters at that point, right?
It's going to take time for the models to, to come back with the right answers.
Now, if, if you run a fine tuning step before that.
So if you can produce long, accurate chain of thoughts for those math equations,
for example, and I'm picking math because that was in the paper, then the model
is gonna look at that and say, okay, I'm doing this equation here, and you
explain every single step, step one, step two, step three, and then I'm
reflecting back, this was right, this was wrong, and then finally I'm gonna
check the answers back, then you're just going to need less steps for the model
to be able to learn what it has to do.
And then you can use RL afterwards to go, oh, this particular
song- sum you went wrong.
So I'm going to give you a cookie here.
Um, because you've now, you know, we've now given you a different way, you
know, here's the right way of doing it.
And you don't get a cookie if you got it wrong.
So I think, I think that combination between the two is the key thing.
But I actually think the real, uh, take away from that paper
is the long chain of thought.
So when I did my experiment, uh, on my YouTube channel, the thing that
I did is I took a slightly different approach from what DeepSeek did.
And, and I have a thing called a math compiler.
So what I did is I, I automatically generate the, uh, the math equations,
and then I put it into my compiler, and I generate an abstract syntax tree, and
then I walk the tree, and then it, I don't need the LLM to do the math, I'm
just going step zero, step one, step two, and I'm just walking the tree, and I'm
outputting the explanations, and then what I do is I use the LLM to transform that
into something that the model actually unders- you know, is actually human
language, and the explanations behind it, and then that's how I got these
really accurate, uh, chains of thought.
And then when I put that in just as an, uh, you know, fine tuning step, I think
I used maybe a hundred different examples and and honestly the math and I did it
on a one and a half billion parameter model the math was incredible, right?
It was like a couple of decimal precisions of uh, you know accuracy out which which
the larger models of six months ago would be nowhere near so so I think the this
The real innovation is the long chain of thought and the accurate chain of thought.
It's not to say RL won't get you there, it will get you there, but
it's just going to take a long time.
So if you can, if you can short set that a little bit, and then have RL
sort of, you know, do the kind of, uh, the smooth and out of the edges,
then you're really going to win.
That's kind of my view on this.
RL is really valuable for tasks like math and things where it's
easy to check the accuracy, right?
Um, as well as, relatively easy to generate that chain of thought, but when
we look like in the paper, for example, they talked about still needing some
instruction tuning for tasks like tool calling, you know, instruction following,
like there's still going to be a need for having like these reasoning models
aren't designed to do every single task.
They're specific for reasoning, and you're still going to potentially
need instruction tuning in order to handle some of those more specific
instruction following tasks.
So I'm going to move us on to our next segment.
Uh, so this is super helpful, I think, in terms of setting the scene, knocking down
some of the myths that have popped up.
We've already talked a bunch about distillation, um, and I think on
the last episode, Skyler actually gave like a short, brief explanation
of it, but for those who weren't listening on the previous episode,
maybe Aaron, I'll toss it to you.
I think it's worth it for our listeners to, just get a sense of like, what
is distillation in the first place?
Um, and then I think if you want to give that explanation, there's some interesting
things I think that are worth getting into about like, well, what does this
mean for where the industry is going?
But maybe I'll toss it to you to give the quick capsule explainer first.
Yeah.
So, I mean, I mean, model distillation is very powerful technique.
You know, it's about having a teacher model, you know, that could be a bigger
model where it's encoded, you know, much more information, uh, through
weights and through embeddings.
And what you want to do is transfer that knowledge to a student model.
And then usually that student model could be smaller and it requires, then in turn,
less resources to train, right, and to also use for inference, right, and some
people and groups think of this as model compression, right, where you're making
a model smaller, and so on, um, and then, and then there's, there's different things
that you can distill, right, you can distill like response based knowledge,
You can dispel feature-based knowledge or even like the relations between all
the different connections within all of the neurons that you have, right?
And one interesting thing that I wanted to bring up that I saw within
the R1 paper is that the distillation process, it wasn't just about, to me,
about just doing this, you know, model compression or getting knowledge out.
But it was almost like this model translation.
Uh, because what I saw is that you were actually distilling
information, uh, from an MOE, right?
And then you were going directly to the student model, which was a
densely connected, you know, feed forward, uh, neural network and
many, uh, different cases, right?
And so, and so just changing that model architecture, um, looked to be,
um, a different way of doing this type of model distillation that, that I
thought also looked You know, gave R1, I think, some advantages, especially
whenever you were looking at using like Qwen2.5 and like the Llama 3
series, right, as the base foundational model to pull information out.
Yeah, and I think one of the most interesting elements of distillation is
sort of the idea that um, you know, you, you can, you can take any large model
and bring that sort of knowledge into whatever it is that you're building.
Um, you know, I think really just literally, I think in the last 24, 48
hours, there was a little bit of kind of a controversy over did DeepSeek use
effectively kind of OpenAI's chains of thoughts or other inputs, outputs
to kind of do the distillation here.
Um, I guess, Kate, kind of question is like, this makes it very hard for
any model company to kind of protect its models in some ways, right?
Because everything is distillable.
Is that the right way of thinking about it?
Yeah, I mean, I think by releasing a very capable, the most capable model
to date in the open source with a permissible MIT license, DeepSeek is
essentially allowing and eroding kind of that competitive moat that all the big
model providers have had to date, keeping their biggest models behind closed doors.
And regardless of whether or not DeepSeek also benefited from distillation from
those bigger models, we're now able to go and take that really big model in the
open and use it indiscriminately, where before people, I mean, this distillation
from GPT has been going on for, ages.
Anyone can go to Hugging Face and find tons of data sets that were generated
from GPT models, uh, that are, you know, formatted and designed for training and
likely taken without the rights to do so.
Um, so this is like a secret that's not secret that's been going on forever.
So yes, it most likely worked its way in some degree or fashion to the
DeepSeek model, but it almost doesn't matter anymore because DeepSeek now is
out there and that model can be used to run very similar style distillation.
With great effect on as many small models as you like, and anyone now has
the rights if they, uh, use DeepSeek's model to do so, according to the
license that it's published under.
That's right.
Yeah, I think one of the funniest parts about the kind of news cycle has been
like, they used a secret, you know, sinister technique called distillation.
Yeah, it's like, actually everybody's been, everybody's
been distilling all the time.
It's just like happening
It's been around forever.
And it costs 5.5 million dollars.
That's right.
Yeah, exactly.
What strikes me, I mean, Chris, even to the example that you gave earlier,
right, like you don't, it turns out you don't need a whole lot of data to
make these models much, much better.
And it kind of seems like there's this sort of like fundamental thing in the
market where it's like, unless you want to control and really down to the
nth level, prevent people from getting outputs from a model, there's basically
no way to stop distillation, right?
I don't know if you think there's a realistic way to prevent that at all.
No, I don't think so.
I mean, the reality is, as Kate said, there's open weight models out there,
um, and people are gonna do that.
And I, I think that, and, and I love this by the way, and the reason I love
it is that I'm, I'm all for chaos.
I'm all for open source.
I'm all for sharing and collaborations.
So, you know what, people are going to go off now, they're going
to create their own data sets.
They're going to distill from different models.
They're going to share that out in the community.
And you know what, we're going to all end up with better stuff.
Right?
So I'm, I'm not a big fan of the closed models, personally, my opinion.
Um, I'm a big fan of sharing and learning from each other.
So I, that's what gets me excited about kind of the DeepSeek stuff.
And, and it, and again, it's not just the fact that, um, they put the model out
there that you can distill from is they talked about the techniques that they use.
So, so yeah, it's, it's cool.
We can all start doing interesting things.
And, and you know what?
I don't think everybody's going to, I don't think we're suddenly all
going to be going out competing with OpenAI, Anthropic, blah, blah, blah.
I don't think the, you know, all of these people sitting in their
bedroom are going to do that.
But you know what they might be able to do is take one of these out of the box
pre-trained model and then solve one of their own particular tasks that the
general model can't do that's specific to their use case and make it easier.
Um, but again, don't, don't undersell this.
I mean, Kate, you know this better than anyone, right?
Fine tuning models is really hard, right?
Because of all of the biases, you might, you might think, hey, my model is now
great at doing this one particular task.
But then you've just ruined that model from doing any other tasks there
because you didn't have the right biases and mixes within that data set.
Yeah.
I mean, just take a look at the Hugging Face Open LLM Leaderboard.
All those distilled versions of Llama and Qwen are on there, and they all rank
significantly lower than the original model that they were distilled from.
Uh, on those Open LLM Leaderboard tasks, which are not predominantly
reasoning-based tasks.
So the model was boosted in reasoning, but other general
performance characteristics drop.
But I, I think it's still incredibly powerful.
And as we talk about, you know, DeepSeek's introducing this new
era of efficient open source AI.
It's true.
It's just not true because they trained this really cost-effective
model during the pre-training.
It's true because we now have the methods to create these armies of distilled fit
for purpose models that are specific for the tasks that you care about because we
have better tooling, like powerful teacher models, out in the open source ecosystem.
Yeah, yeah, yeah.
I think that there's a lot of secret agents, you know, that
are hidden amongst our labs.
And, you know, in the next couple days, weeks, you'll see them to
become super agents that are going to be a release that we can all use.
So, um, I, I really think this might have been one of the impetuses, you know,
to sort of grandstand out, you know, what's happening within the field of AI.
Um, DeepSeek just happened to be right time, right place, you know, to do it, to
put all, to connect all the dots together.
Um, but, yeah.
Um, I do think that lots of these technologies and new innovations that
are coming out, inventions, um, you know, you ask the question, can you
prevent someone from distilling a model?
Um, that sort of brings me back to biometrics.
You know, it used to be, can you prevent someone from
stealing a picture of your face?
You know, and we, and we came up with this cancelable biometric
invention so that if someone took your picture, you could revoke your
biometric and create a new one, right?
So I mean, I mean, there, I think there might be some cancelable
technologies, um, and patents, right?
That we could work on together, right?
To achieve some of this.
Final question here, I think for Kate, um, particularly given your work on
Granite, you know, I think there's maybe one point of view, which is, well,
you know, the only reason you know, investors have put money in towards
building these giant, giant models is kind of the idea that if you build
these giant models, you'll be able to capture all the value from that model.
And it sort of seems to me that like, if distillation gets good and you know,
granted, distillation is hard in some respects, but you think you've got
enough eyeballs, someone will eventually figure out ways of cracking it.
You know, is there an argument here that it kind of erodes the incentive
for people to invest in building the big model in the first place?
Uh, like, and there's kind of a really interesting question, which is, you
know, it's almost kind of an accident that like we've ended up with these
giant models and it's partially based on the idea that like, well, you could
have some exclusive control over this, but it feels like this is rapidly kind
of escaping the ability for anyone to be able to kind of exclusively control.
Yeah, look, I don't think there's any incentive to really build big
models to run at inference time.
The incentive is to big, really big models to help you build really small models.
And all it takes, like we've, it started with Llama releasing, you
know, 400 billion plus parameter model, NVIDIA released a 400 billion
plus parameter model as a teacher.
And now DeepSeek releasing their 600 billion plus parameters,
you know, size isn't everything.
They also have to have high quality post-training, which is
why the reinforcement learning part of DeepSeek is so important.
But we're seeing more and more large models that can be used openly
to train these smaller models.
And I think it's just going to continue to make this more of a teacher model
based commodity like why pay for those big models if we've got similar capabilities
out in the open that you can customize further and I think we are going to
converge on to a point where we've got powerful enough tools to craft the smaller
models that we need that are going to run, you know 80 percent to 90 percent of our
workflows for generative AI in the future.
Yeah, it's kind of a funny world where it's kind of like you ever, you
never talk to the giant model that's just inside company headquarters and
then just like lots of tiny models that are coming out around it.
Well, in the last few minutes, I want to zoom out a little bit.
We've been talking a lot about DeepSeek and what's going on underneath the hood.
Um, and I want to just take a moment to talk a little bit about what
all the other companies are doing.
Relative to this development in the AI space, um, Sam Altman, of course, the
head of OpenAI put out a little tweet thread kind of responding to this news.
Um, and I'll just quote a little bit of it.
He said: "we are excited to continue to execute on our research roadmap and
believe more compute is more important now than ever to succeed at our mission."
Um, which is really like a statement by a guy that says, steady as she goes, we're
continuing on the research path as you know, we had planned, um, and nothing
has changed by the DeepSeek, uh, release.
Um, I guess, Chris, maybe I'll kick it to you.
Do you buy that?
Like, is OpenAI pretty much gonna just keep doing its strategy?
Or does this really kind of fundamentally change what they're gonna need to do?
Nah, he's gonna release his model sooner.
He's been holding on to these models for too long, and he needs to get on with it.
And good on you, DeepSeek, right?
Where's my o3?
You showed me it at the end of Christmas.
Do I have it in my hands?
No, so thank you DeepSeek, maybe we'll get his model out a bit quicker and
then we'll get o4 and o5 and then maybe we'll get some of these models
in Europe because guess what, they're releasing vision models and video
models and I don't have any of them so I'm gonna get them as well, so woohoo!
Uh, so I guess ultimately what you're saying is it just
accelerates his roadmap, right?
To just get him off the fence.
There is no way he's just gonna sit there and go, uh uh uh, I'm not
giving you my model while DeepSeek is getting all of this press.
He's gonna respond and we're gonna get new models.
But I think, I mean, Aaron, maybe to turn it to you, you don't think this
changes, like, their approach to, I guess, ironically, being kind of
a closed source model here, right?
Like, this is not the kind of situation where you believe that OpenAI or
Anthropic, any of the big kind of providers, would say, hey, now we
need to start switching to open source is the way we play this game.
Um, I don't think so.
I mean, I mean, I think so.
I mean, this, this could go in several directions, you know, but I
think, you know, open versus closed source, you know, I think that there's
advantages and disadvantages to both, but I think ultimately it helps the
academic community, which then in turn fuels, you know, economies of
scale for the average consumer, right?
Because, uh, because if you think about it, you have two groups, right?
You have the open source group, closed group, they compete, you know, um, to
make sure that one is better than the other, which then spurns innovation.
Okay. Great.
And then within each one of those groups, you have companies and
organizations that then in turn competes.
You have like this 20 and of competition that further accelerates,
you know, this, uh, innovation.
And so I, and so, so Sam Altman, you know, you know, I think he's going to release a
secret agents, right, uh, sooner, right.
Make them available, right.
And, and lots of the techniques that, you know, DeepSeek has shown, you know,
like that caching layer of the key value and queries that they've, you know, come
up with some of their MoE innovations and then some of the parallelization
whenever they can share context and information amongst their grid, lots of
that is going to be included, I think, in Sam Altman's but pushed even further,
you know, with their own innovations and it's going to splinter out a bit, but
the fundamental like model distillation and so on and so forth, you know, I
think that's going to be, uh, very key.
And then it brings a value proposition down to frameworks, you
know, um, how can I better train the models for my own fit purpose?
You know, whether I'm an enterprise or a customer and then
also how can I trust it, right?
Because there's going to be a zoo of models now that are out there.
It's just very confusing to pick which one do I use?
Yeah, Kate, uh, so we've been talking about OpenAI.
Obviously they take up a bunch of sort of airtime.
Um, but I guess one thing to kind of, as we think about zooming out to tell this
DeepSeek story is whether or not we think OpenAI is kind of similarly situated.
Like, you know, everything we've been hearing is, okay, OpenAI is
going to continue its strategy, it's just going to move faster.
Do you think it changes the economics at all or the kind of decision making
at all for say a Google or a Meta or, um, you know, even like an Anthropic?
I don't think it changes the decision making or strategy, uh, overall.
I think a lot of DeepSeek strategy, you know, necessity
is the mother of invention.
They only had access to H800 chips.
So they optimized the hell of it.
They invested in efficient architecture like MoEs and DeepSeek was born, right?
So I think the U.S. based labs are operating with very different constraints
and DeepSeek's innovation doesn't necessarily, constraints and incentives.
And I don't think DeepSeek necessarily changed that calculus.
I also think a lot, again, what we've talked about today
with DeepSeek is distillation.
And for the labs pursuing AGI, distillation is not
necessarily as relevant, right?
They need to keep training as big a model as possible and have incentives to
try and keep that behind closed doors.
Whereas the business value, again, my take is the business value is all
around these distilled smaller models that are actually what people are going
to deploy in a commercial setting.
And I don't think they're at least at the highest strategy level
and what they're working on with right in terms of their investment
profiles, that longer term AGI game.
And for that you still need a crap ton of big model of big GPUs.
And they're not going to want to release any of that out in the open.
Right?
Yeah.
It's not like they're going to use Stargate to do like
small distilled models.
That would be the funniest thing.
It's actually an inference cluster.
Surprise.
Um, that's yeah, that's I think really fascinating.
I guess, Chris, maybe I'll turn to you for the sort of last
word and last question here.
You know, Kate just talked a little bit about the idea that
Chinese researchers are operating under very different constraints.
So they kind of develop different types of methodology, different types of
models, different types of proficiencies.
Um, and do you think there's something to the idea that like, almost we're
like, we have an embarrassment of compute
among the U.S. labs.
And
So it actually kind of like, limits, like the degree to which we would
ever invest in the kind of thing that DeepSeek would be working on.
Um, I'm really sort of interested in the idea that like, these constraints really
mean that AI will start to look pretty different in different parts of the world.
As researchers operate under very different constraints of what
they need to do to deploy systems.
I, I think that's exactly the case, right?
And you can see a little bit of reinforcement learning happening
there and reward modeling, right?
Which you, you were saying here.
You're going to have less compute available to you.
And guess what?
They have different incentives at that point, and they've been
rewarded by being more efficient.
So if you've got an abundance of compute, you're not really going
to be optimizing for efficiency.
You're going to be trying to be getting your models out first.
And I think that's also, you know, speaking from my own
experience, I, you know, I don't
have any H100's kicking
around.
What have I got?
I've got my MacBook Pro, right? Where I've got -- You're more like the deep sea researcher, basically.
Exactly.
So you're trying to come up with innovative techniques to work
within the hardware constraints that you run within today.
So I, I, and I think honestly, if they didn't have the chip constraints in China,
I'm not sure that DeepSeek would have probably came up with those techniques,
because they, they would have been just trying to focus and catch up with
everybody else, as opposed to trying to take things from a different angle.
And, and therefore, again, one of the reasons I believe in open
source very much, and everybody's sharing their papers, everybody's
running under different constraints.
And they're going to find new innovations, and if we share that,
we're all going to learn from each other and be able to contribute.
And that's not just the big labs, but the, the people in the community,
just with their laptops, trying to discover and experiment with new things.
Ah, so I love this panel.
Kate, Aaron, Chris, thank you for joining us on the show as always
and walking us through DeepSeek.
Uh, a lot more to talk about and we will be tracking the story.
Um, and thanks for you listeners for joining us.
Uh, if you enjoyed what you heard, you can get us on Apple Podcasts, Spotify,
and podcast platforms everywhere.
And we will see you next week on a jam packed episode
again of Mixture of Experts.