Training vs Inference Hardware Landscape
Key Points
- The episode focuses on how the training and inference hardware stacks are increasingly diverging, raising challenges for designing datacenter‑grade chips that remain viable for 5‑6 years as model architectures evolve.
- Apple’s hybrid approach—running simple tasks on‑device and off‑loading more complex reasoning to the cloud—is highlighted as a potential industry‑wide pattern for improving composability of chips and models.
- Model‑optimization techniques are becoming more accessible, with growing tooling and documentation that help developers squeeze maximal performance out of existing hardware.
- NVIDIA’s recent dominance in AI hardware investment is noted, but speakers emphasize that differing consumer needs and use‑case requirements are driving a split between training‑focused and inference‑focused ecosystems.
- Guests include Volkmar Uhlig (VP of AI Infrastructure), Chris Hay (CTO of Customer Transformation), and Kaoutar El Maghraoui (Principal Research Scientist for AI Hardware), who discuss these trends and their implications.
Sections
- Hardware Trends & Apple Hybrid - Brian Casey explores the split between training and inference hardware, the longevity challenges of data‑center chips, Apple’s on‑device‑plus‑cloud model, and emerging model‑optimization techniques.
- Scaling Inference and Training Across GPUs - The speaker outlines how inference is limited by latency, throughput, and memory—necessitating multi‑GPU (A100/H100) configurations to keep token generation human‑readable—while training emphasizes connecting thousands of GPUs and maximizing inter‑GPU communication speed to achieve cost‑effective performance.
- Clients Prefer SaaS Over On-Prem GPUs - Clients avoid managing their own GPUs for inference due to cost inefficiency, opting for token‑based SaaS services, while only regulated sectors require on‑prem GPUs, making cost reductions vital for startups and SMEs.
- Debating LLM Dominance and Alternatives - The speaker reflects on Jan Leike’s warning against a singular focus on large language models, asks whether hardware players are similarly fixated, and notes ongoing research into alternative model architectures.
- From Paper Hype to Industry Impact - The speaker reflects on the viral excitement surrounding a new research paper, the gap between initial hype and practical reality, and outlines the milestones required to transform the novel approach into commercially relevant hardware.
- AI Model Lifecycle & Hardware Shift - The speaker explains that large language models typically have a 5‑6‑year relevance window, with successive 10× performance gains enabling ever more complex architectures, while hardware manufacturers are rapidly re‑optimizing GPUs for the memory‑bandwidth‑intensive transformer stack, shifting from many‑GPU setups to fewer, more specialized cards.
- Consumer Hardware Constraints vs Model Innovation - The speaker explains how consumer‑grade hardware limits model deployment, weighing the choice between sticking with established CNN hardware and adopting large, fast new architectures, and the resulting challenges for hardware vendors.
- Stakeholder-Driven Factors in AI Deployment - The speaker outlines how consumers, model‑training companies, and cloud inference providers each prioritize distinct concerns—cost, hardware compatibility, training speed, and token‑cost efficiency—shaping model design and the growing importance of composable model architectures.
- Secure Edge-Cloud Architecture Discussion - A speaker outlines a client‑focused architecture that extends trust from a handheld device into a data centre, emphasizing secure boot, non‑introspectable admin access, digitally signed binaries, and physical isolation.
- Apple M3 Enables Local LLM Fine‑Tuning - The speaker explains that the M3’s 128 GB unified memory allows them to run and fine‑tune massive models such as Llama 3 70B directly on a MacBook, eliminating the need for cloud resources and positioning Apple as a developer‑focused AI workstation.
- LLM Optimization: Techniques & Tools - The speaker highlights how combining fine‑tuned small models with compression methods like quantization, pruning, and knowledge distillation—supported by libraries such as Hugging Face Optimum—delivers lower inference costs while maintaining performance, reflecting the current market excitement.
- Pre‑tokenization and Feature‑Based Freezing - The speaker explains how pre‑tokenizing data and batching by equal lengths dramatically speeds up fine‑tuning, and predicts that future optimizations will move from freezing whole layers to selectively freezing individual learned features.
- Rapid Gains in Mixture‑of‑Experts - The speakers wrap up a discussion on continuous batching, noting its low latency and unchanged quality, while celebrating the excitement of sudden 10× performance improvements in the rapidly evolving Mixture‑of‑Experts field.
Full Transcript
# Training vs Inference Hardware Landscape **Source:** [https://www.youtube.com/watch?v=rn1sjMFRzTQ](https://www.youtube.com/watch?v=rn1sjMFRzTQ) **Duration:** 00:38:18 ## Summary - The episode focuses on how the training and inference hardware stacks are increasingly diverging, raising challenges for designing datacenter‑grade chips that remain viable for 5‑6 years as model architectures evolve. - Apple’s hybrid approach—running simple tasks on‑device and off‑loading more complex reasoning to the cloud—is highlighted as a potential industry‑wide pattern for improving composability of chips and models. - Model‑optimization techniques are becoming more accessible, with growing tooling and documentation that help developers squeeze maximal performance out of existing hardware. - NVIDIA’s recent dominance in AI hardware investment is noted, but speakers emphasize that differing consumer needs and use‑case requirements are driving a split between training‑focused and inference‑focused ecosystems. - Guests include Volkmar Uhlig (VP of AI Infrastructure), Chris Hay (CTO of Customer Transformation), and Kaoutar El Maghraoui (Principal Research Scientist for AI Hardware), who discuss these trends and their implications. ## Sections - [00:00:00](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=0s) **Hardware Trends & Apple Hybrid** - Brian Casey explores the split between training and inference hardware, the longevity challenges of data‑center chips, Apple’s on‑device‑plus‑cloud model, and emerging model‑optimization techniques. - [00:03:02](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=182s) **Scaling Inference and Training Across GPUs** - The speaker outlines how inference is limited by latency, throughput, and memory—necessitating multi‑GPU (A100/H100) configurations to keep token generation human‑readable—while training emphasizes connecting thousands of GPUs and maximizing inter‑GPU communication speed to achieve cost‑effective performance. - [00:06:07](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=367s) **Clients Prefer SaaS Over On-Prem GPUs** - Clients avoid managing their own GPUs for inference due to cost inefficiency, opting for token‑based SaaS services, while only regulated sectors require on‑prem GPUs, making cost reductions vital for startups and SMEs. - [00:09:13](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=553s) **Debating LLM Dominance and Alternatives** - The speaker reflects on Jan Leike’s warning against a singular focus on large language models, asks whether hardware players are similarly fixated, and notes ongoing research into alternative model architectures. - [00:12:17](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=737s) **From Paper Hype to Industry Impact** - The speaker reflects on the viral excitement surrounding a new research paper, the gap between initial hype and practical reality, and outlines the milestones required to transform the novel approach into commercially relevant hardware. - [00:15:20](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=920s) **AI Model Lifecycle & Hardware Shift** - The speaker explains that large language models typically have a 5‑6‑year relevance window, with successive 10× performance gains enabling ever more complex architectures, while hardware manufacturers are rapidly re‑optimizing GPUs for the memory‑bandwidth‑intensive transformer stack, shifting from many‑GPU setups to fewer, more specialized cards. - [00:18:25](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=1105s) **Consumer Hardware Constraints vs Model Innovation** - The speaker explains how consumer‑grade hardware limits model deployment, weighing the choice between sticking with established CNN hardware and adopting large, fast new architectures, and the resulting challenges for hardware vendors. - [00:21:38](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=1298s) **Stakeholder-Driven Factors in AI Deployment** - The speaker outlines how consumers, model‑training companies, and cloud inference providers each prioritize distinct concerns—cost, hardware compatibility, training speed, and token‑cost efficiency—shaping model design and the growing importance of composable model architectures. - [00:24:54](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=1494s) **Secure Edge-Cloud Architecture Discussion** - A speaker outlines a client‑focused architecture that extends trust from a handheld device into a data centre, emphasizing secure boot, non‑introspectable admin access, digitally signed binaries, and physical isolation. - [00:27:57](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=1677s) **Apple M3 Enables Local LLM Fine‑Tuning** - The speaker explains that the M3’s 128 GB unified memory allows them to run and fine‑tune massive models such as Llama 3 70B directly on a MacBook, eliminating the need for cloud resources and positioning Apple as a developer‑focused AI workstation. - [00:31:15](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=1875s) **LLM Optimization: Techniques & Tools** - The speaker highlights how combining fine‑tuned small models with compression methods like quantization, pruning, and knowledge distillation—supported by libraries such as Hugging Face Optimum—delivers lower inference costs while maintaining performance, reflecting the current market excitement. - [00:34:21](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=2061s) **Pre‑tokenization and Feature‑Based Freezing** - The speaker explains how pre‑tokenizing data and batching by equal lengths dramatically speeds up fine‑tuning, and predicts that future optimizations will move from freezing whole layers to selectively freezing individual learned features. - [00:37:25](https://www.youtube.com/watch?v=rn1sjMFRzTQ&t=2245s) **Rapid Gains in Mixture‑of‑Experts** - The speakers wrap up a discussion on continuous batching, noting its low latency and unchanged quality, while celebrating the excitement of sudden 10× performance improvements in the rapidly evolving Mixture‑of‑Experts field. ## Full Transcript
Welcome everyone to this week's Mixture of Experts.
I am your host, Brian Casey.
Um, today we're doing all hardware all the time.
Uh, we are going to start with a deep dive really of the industry
and space right now, specifically about, um, how the training and
inference stacks are diverging.
Model architectures still.
evolve over time.
So how do you build hardware that, you know, you put in the data center and you,
you know, depreciate over like five, six years so that you can still run the model,
which is active in six years with today's hardware, you know, you're putting down.
From there, we're going to move to Apple and talk a little bit about how The
architecture patterns that we see there that are combining on device and cloud
could be a pattern for the industry.
The composability of these chips, the different models
is going to become important.
And I liked the way Apple did this, right?
Here's some of the stuff that you run on device.
But when I want to do something a little bit more complicated than
reasoning, I'm going to come off device and I'm going to push that
on the cloud to perform that action.
Finally today, we're going to talk about model optimization and the things
that people are doing with models.
to better take advantage of the hardware that's available to them.
But I see there's a lot of energy there and a lot of work and documentation
and, uh, which make it very easy, you know, for the end users and the
developers to use these techniques.
Joining us on the show today, we have Volkmar Uhlig, who is
the VP of AI Infrastructure.
We have Chris Hay, who is the CTO of Customer Transformation and we have
Kaoutar El Maghraoui, who is the Principal Research Scientist for AI Hardware.
We're going to start like a little bit big picture.
Um, and Volkmar, I'm going to throw this first one over to you.
Um, but obviously like NVIDIA has been, has absorbed a lot of like the oxygen
and perhaps money in the room, uh, in the hardware space over the last 18
months in a sort of like classic, Um, when there's a gold rush picks and
shovels, um, toward a sort of investment.
Uh, but I think one of the thing that's becoming like more discussed and kind
of more clear is that the ways that like the training and inference stacks might
diverge, um, over time, the extent to which that's kind of already happening.
Um, because I think some of the players that are involved in each
one of those things, and some of the like, the interests of the consumers
are, um, you know, a bit different.
Yeah.
Even at some of the use cases obviously are very different, but I wonder if you
might just like start us off and just like maybe give, like, I don't want to say an
overview of the whole landscape, but maybe start with that kind of piece of talking
about the, like the two stacks of like training and inference, uh, kind of maybe
some of what you see going on in each of those and just like how you see them
kind of diverging both now and over time.
Yeah.
Okay, cool.
So I think the, uh, uh, the main difference between.
inference and training stack is to scale, uh, these GPUs get connected.
And so we like, there's a bunch of accelerators happening, but
let's just stay in the GPU market.
On inference.
You're primarily limited by, you know, the throughput you want to
achieve the token latency you want to achieve, um, and then there's a
certain amount of memory you need.
And so the bigger the models get, then usually, you know, you start
spanning across multiple GPUs just to maintain, like you have enough
memory and maintain the throughput, um, and then achieve latencies, which
are reasonable for human beings.
So if you look at the big GPUs, which is often what...
what people are using, A100s, H100s, and, you know, what's coming out, um,
you connect, you know, two to four of them to just get into latency bands, um,
where, uh, you know, the human can not read as fast as the tokens get produced.
Um, and then you have the, the other area, um, we're just training and training is
really, uh, how many GPUs can you connect into a single system, and it's much
more like the HPC space, you know, very like, you know, for 20 years, we tried
to figure out how to build networks to connect thousands or tens of thousands of
computers, and now we're doing the same thing again, it's just called the GPU.
And here it's really how fast can you exchange information between these GPUs?
So you have these all gather operations or weight redistribution operations, um,
but you may have 10, 000 GPUs behind it.
So the real value here is how fast can you get data out of these GPUs
to the other GPUs and how fast can you distribute them to all.
And so from a cost perspective, um, in, in basic inferencing is really like, you
know, you cram something in a box and you want to have a power efficient, um, and
then you want to achieve those latencies.
And in the, in the case of training, uh, at least for like the very large
training sets, uh, it's like how many computers can you fit into a
room and connect them with a super latency, uh, super low latency network?
And many cases like You know, the hardware cost in the network is
30 percent of your overall system cost just to interconnect them.
And Chris, maybe a question to you is, um, on the inferencing side, like in
particular, have, have you seen like costs on like an absolute basis already become
like a real issue for clients, or is it the type of thing where, you know, they're
concerned about like the unit economics and as it scales, it will obviously
be an issue, like how real is the cost problem today um, on just like the
production workloads people are working with when it comes to work with LLMs?
Is that like a theoretical problem or real problem?
Um, a very obviously in the future problem, uh, real problem, but how
would you, how would you describe that?
I honestly think from a client perspective, and this is going to sound
really harsh, they don't really care.
Because most of the time they're getting inference from the cloud, right?
And they're playing cost per token, so they're not really provisioning GPUs.
I don't know many clients who are provisioning GPUs?
Maybe some are doing that for fine tuning, but they're certainly
not doing that for inference.
In fact, clients are actively avoiding anywhere where they have to have
a GPU for inference themselves.
And the reason they're avoiding that is they're not going to shove enough
workload through that for inference to be able to justify the cost.
So they really want to be paying more in a kind of SaaS model
where it's kind of paying for the tokens as opposed to GPU time.
And if you think about it sensibly, GPUs would just sit around doing nothing most
of the time in a client's estate, so I think this is a big technology company
problem, not necessarily a client problem.
Now, don't get me wrong.
There are some clients who have to run on premise workloads.
So think of financial institutions, government institutions where they don't
want that data going to the cloud, and they have that problem where they have
to think about provision their own GPUs.
But, but regular clients, I don't think they think about that too much.
But I think I, you know, this is very true, Chris, but I think especially
for startups and like small and medium businesses, this becomes a big challenge,
especially being able to maybe either train or fine tune models for their
whatever business that they care about.
This becomes a real issue.
So cost reductions for them will be significant.
I think like what Chris said is, is right.
To a certain extent, the customer doesn't really care.
Um, you know, they don't want to know that there's a GPU behind it, ultimately.
Um, they only care that they get the service.
And they get the service at a certain cost and at a certain latency.
Because in the end, you're modeling something where, let's say, I have an
interactive workload with a customer and the customer doesn't want to wait like
20 seconds until they get an answer back.
Um, we want to hide that in the data center in the inferencing service.
Now, from a hardware perspective from, you know, companies which are
actually invested in those hardware systems, that's the optimization, right?
And that's, I think, where we will see much more custom hardware going
into the market, um, which is highly optimized for these specific workloads.
And it's interesting to see like, you know, we have a trend of like, okay,
there's, you know, more common models and model architectures, but model
architectures still evolve over time.
So how do you build hardware that, you know, you put in the data center and you,
you know, depreciate over like five, six years so that you can still run the model,
which is active in six years with today's hardware, you know, you're putting down.
So I think there is a, there's an optimization game probably played by
the big players, you know, the AWSs and you know, the Azures and you know,
the IBMs who are putting specialized hardware, highly optimized influencing
engines with a lot of variety for these different models into the data center.
And I think, and for companies, it'll be hard because, like,
how many GPUs do you wanna host?
Like you want to homogenize, but now you may overpay.
And so the cloud may be your answer, uh, to get cost reductions in place.
To that point about maybe optimizing for five or six year time horizons.
Like one of the themes I've seen come up, um, more recently,
I've heard Jan Leike....
Um, make versions of the statement, a few others, um, that like LLMs,
even within the research and academia community have like absorbed all of
the oxygen, um, in, in the room and that there's other, there's other
areas and like ways that people could potentially even imagine getting to AGI.
But like right now, it's just everybody's on this one path and so I
think Jan even like famously was like telling New Research like don't work
with LLMs like go do something else with your life, uh, basically which
like the reaction in the market was like sort of predictable to that but
it was like very Jan on some level.
Um, but I'm curious like do you see like the hardware players like almost following
that same sort of dynamic where like everybody sort of like is looking what's
happening with LLMs right now and they're just sort of all in like the transformer
has been pretty resistant up to up to this point with some like tweaks to it so...
are people kind of looking at the model architectures that are around today and
just sort of like assuming that those are going to hold for at least at least
the medium term or, you know, do you see places where people are like, you
know, Exploring, you know, what other, um, um, you know, maybe what other
alternative architectures might play a role at some point in the future?
I think that's a great question, and there's a lot of research
actually exploring alternative architectures as transformers.
Transformers, of course, it's a pretty solid architecture.
Uh, the example, for example, the MatMal3 LLMs.
Which is… which is I think it’s a WAG paper that has demonstrated the viability of MatMal3 architectures.
And the research opens up a new frontier in LLM design.
Also encouraging the exploration of alternative operations and computational paradigms.
So this is actually kind of opening the frontier here for a new wave
of innovations with researchers developing novel model architectures.
So this particular paper, Uh, really kind of looked at using these trinary
operations, replacing the, uh, the math behind the matrix multiplication with
much simpler operations, and that's actually the biggest bottleneck you
see in these LLMs computationally.
And there's also translated also into reduction, huge reductions into the memory
and the computational, uh, requirements needed for these LLMs, also, they really
showed some very interesting benchmarks and result that shows you could get almost
same accuracy or the same performance as you would get with transformers.
So and I think this is just the beginning.
So there is a huge need to look at alternative architectures, uh,
either using, you know, these novel architectures or using things like,
uh, neural architecture search where you're trying also to You using AI
to generate efficient architectures for for hardware for to target, you
know, special special platforms.
And that's are much more energy efficient.
The reaction to that paper when it came out in the market
was like pretty substantial.
Um, I think the original tweet that got shared as well, like announcing the
paper, either just like summarizing it even, um, I think had a couple million
views associated with it and people were very excited and, um, in the way
that Twitter gets, um, around things like, oh, this is, you know, a big
change, it's like, we're not going to do matrix math on GPUs, uh, anymore.
And obviously, like there's the initial hype and then, you know, there's
reality that sort of comes after that.
Um, but in, in a space like that where you're doing that sort of
fundamental research and a lot of the indicators were looking like, you
know, this actually could be a thing...
um, what do you, what are like the next steps for an approach like that going
from something that's just like a novel piece of research to something that
is like approaching a place where it might have some commercial relevance?
Like, how does it go, and like, you know, does it end up having any
impact on the hardware ecosystem?
Maybe before we get to that hardware point, you know, maybe just would love
to hear from the group of, you know, what are the things that you would want to
see that would give you some indication that like a different approach like this,
um, might have some, some relevance and some staying power, um, in the industry?
So I think that we go through functional plateaus with these different models.
So if you go back, we had CNNs, and suddenly computers could see.
And then we got transformers, and suddenly they can reason.
And we are going through that, through those phases, I think there will be new
model architectures coming, you know.
Like this is almost like a scientific discovery.
Suddenly, The model exhibits a certain behavior, which we
haven't seen before, right?
And we had the similar effects when CNNs came out and suddenly, you know, all the
image processing algorithms were out the window and now all the, the NLP systems
are out of the window because we don't do entity extraction anymore because
we can directly reason in a network.
And so I think there will be new architectures coming with new
capabilities, and I think this will be a lot of the effort, like the
people will start experimenting with these alternative model architectures.
At the same time, we are seeing that, you know, people are trying to rewrite
the existing capabilities because now you have that new plateau where you are,
you know, we know how Chat GPT needs to behave and we, we can benchmark it.
Before we could benchmark the vision algorithms and we could say, you
know, what's the detection rate.
And so I believe that we will constantly go through these loops.
And if these fundamental changes happen, suddenly you got, you know, a 10 X
performance improvement, but, and then the, the hardware will catch up, but
you will not take something which is in production and necessarily say, Oh,
let's just throw it out because these models have been tuned for, for ages.
So if you look at the introduction from discovery until it goes into the
industry, you know, if you are like at.
Three years now, like, you know, it's like Chat GPT already I mean, it's already
aged, like the transformer model is aged.
And so what's the lifetime?
It's probably not like twenty years, but it's probably like five, six
years until the new model shows up and things, you know, get tweaked,
and each of these 10 X performance improvements now allow you to make the
model 10 X more complex, because now you suddenly get all the computation free.
And so I think those things will happen over and over again.
We are just now in this era of discovery, right before a true architecture or
a lasting architecture was found.
So, and I think we will have this for a couple more years.
Do you think that basically every, every hardware player in the space right now
is up is like building out what they're doing, optimizing almost entirely for like
the existing transformer and LLM stack?
That's the safe bet, right?
So if you look from a, from a manufacturer, if you build a card,
like you go like all the cards, if you go back five years, cards
don't have enough memory bandwidth.
Then suddenly training of these LLMs comes on online and it's like everybody's
like oh I need two terabytes of memory bandwidth per second and now I need four
and you know if you're growing these cards if you would have optimized the card five
years ago you would have done that for encoder models which are totally compute
bound and not that, not memory bound.
And so we are seeing the shift towards these models as they come online.
And then you go from, oh, I need 32 GPUs to I need one.
But that's the architectural shift because the hardware gets like gets modified.
And you are like, Okay, I'm willing to pay 10 times as much for memory bandwidth.
And so but overall, I'm still getting a better deal.
And so I think those things will happen over and over again, you know, like
effectively what NVIDIA did is like it took the computation of the CPU
and said, okay, I give you 100x for this different architecture, right?
And now we are going through a cycle of, okay, we need more memory.
And so then there will be specialized operations.
We already see it with CUDA cores, right?
So do matrix multiplications in hardware and, you know, you just
do this over and over again.
I, I kind of disagree.
Yeah. Sorry. I kind of disagree. Good.
I think, I think last best model wins.
That's it.
So how many people here is still using LLAMA 1?
How many people here is still using LLAMA 2?
Nobody.
Why?
Because we're using LLAMA 3.
How many people is using Mistral version 1?
Zero.
Cause we've all moved on to Mistral 0.3. Last best model wins.
So therefore, if somebody comes up with a good new architecture that beats the
old architecture, Everybody's going to move to that super, super quickly.
Consumer doesn't care because consumers are not training models, so they
will take the first model that works, they'll run it, as long as it works on
their consumer hardware, they're good.
Slightly different for data centers, but data centers are not going
to want to be left behind, so they're going to run towards it.
So hardware wise, if I'm honest, as long as it runs on my Mac, which
runs MLX, I don't care, right?
So I'm not going to go out and buy a GPU, it's going to run on my MLX.
Now that's different for, uh, folks who run data centers because they're going
to want the most efficient inference that they can and they're going to invest
in the hardware to be able to do that.
But me as a consumer, last best model wins and, you know, it's got
to run on my consumer grade hardware.
Good, but now you just defined what the constraint is and it needs
to run on my consumer hardware.
So there is effectively a, there are two sides of this.
There's like, to what hardware do I build, which you are going to buy
with your next laptop in three years?
And what's the model architecture?
And so you have a deployment, deployment issue, right?
So CNNs, for example, haven't fundamentally changed.
So CNNs, the hardware is the same, right?
It's an encoder model.
It's not a decoder model.
And so you, you, Still run the same hardware, but now we said, Oh, and by the
way, we have this new workloads, but fund fundamentally CNN's all kind of stuck
where they were a couple of years ago
To a point, because if there's a brand new model with a completely
different architecture, and it could be.
It could be a 5 trillion parameter model for all I care, right?
If the latency is fast, then again, I'm putting more constraints on it,
and it's SaaS and I can't run it on my machine, but it smokes whatever
I can run on my machine by 100x, and the price is so cheap, I don't care.
I don't need to run it on my machine at that point, because That model
there is the new best model, and it wins, and therefore me as a consumer
is going to run right towards it.
I think on the hardware space it is, it has been constantly a challenge
for these hardware vendors to figure out what do we optimize for.
There's always this trade offs between building this general purpose accelerator
that can run all these different architectures Or this really super
optimized things for like the LLMs or for the CNN, so for, you know, the LSTMs
or all of these different architectures.
So that I think that's going to continue to evolve and because it takes a long
time to design hardware, so it's not a software that's, you know, every day you
have new architectures, new, new releases and things like that for hardware.
It is a much longer timeframe, so.
so this really, you know, becomes a challenge for the hardware vendors.
What do we optimize for?
And I think maybe that's going to even change the way these hardware
vendors and the architecture and so forth think about hardware moving
into these composable systems.
Maybe I don't need to build these monolithic chips that are built to design
everything, you know, to optimize for one specific model or architectures.
Can we compose things and then plug plug them or compose
decompose in a dynamic fashion?
So I think this is maybe something a chiplet, for example, it's something
trying to do something similar where we we don't want to have, you know,
these one big chip, but having these, uh, uh, chiplets where that you can
compose, uh, very easily and then scale, especially for different use cases.
I think the, the, this race for the best model is going to continue the,
the, also the race on the AI hardware...
how do we optimize, what do we optimize for?
How do we build the next roadmap for the next five years?
That's also going to be a challenge, but it's going to force, I think,
a much closer hardware software co design kind of cycle that we need
to shorten so we can reduce cost efficiencies and win in the market share.
And I think you're right.
There is, I think there's two different driving factors, right?
So if you're a consumer, there are three different driving factors.
If you're a consumer, you care about what is cheap for you, what
can run on your machine, etc.
That's kind of one factor.
If you are a company who is training these models, then the
biggest factor in your case is going to be, can I train the model?
Can I pump the data in, how quickly can I get my model out?
And how can I prove it and keep up with the market?
If you're a cloud company serving up inference or a data center serving up
inference, then you're trying to maximize the cost per token or minimize the
cost per token for your architecture.
And that becomes really important because you want to be
competitive with everybody else.
So if you're charging, you know 0. 05 cents per token or something per million token, but somebody
else is doing it for 0.01, you need to cut your inference costs get it as bare as possible so
that you can be as cheap as possible.
But that's not true when training, Right?
When training you are trying to get the best possible model that you can and
get it done as as quickly as possible.
So I think these different driving forces, uh, really affect it.
And I totally agree with you, Kaoutar, that, that the composability of
these shapes, the different models is going to become important.
And I like the way Apple did this, right?
Here's some of the stuff that you run on device.
But when I want to do something a little bit more complicated than
reasoning, I'm going to come off device and I'm going to push that
onto cloud to perform that action.
And I think that sort of.
Uh, thing is going to be key in the future.
There's a really good statement across which, um, what I like about the Apple
approach is that they effectively figured out an upgrade path, right?
So you can have your five generations of old phones and you effectively
say, whatever I can do on device, I do on device and whatever I can't,
or if I don't have the hardware acceleration, I now have an overflow
bucket and the overflow bucket can be.
The model is too complex or like I want a complex answer to something or
the other one is I have old hardware.
So they effectively created for themselves now an architecture where
they can start innovating despite that the device has a longer lifetime.
So they decoupled the lifetime of the device from the lifetime of
or from from the model evolution.
And so now they can innovate rapidly because they can
always update the data center.
But the device which the customer held in their hand is, it's,
you know, it's kind of fixed.
And so I think it's, it's a really interesting move
to, to do that separation.
I thought that story, just like everything that they announced at WWDC from, from
both that path of smaller, like many smaller models on device to their own
models in the cloud, to only a third party API when they absolutely have to,
and then having like the user opt in on a per interaction basis combined with
their own sort of Silicon was just like...
the best example yet I've seen in the market around how to do, you know, how
to optimize for, for cost for speed, how to do kind of like local, um, you
know, local inference, um, on, on a workload, um, you know, I'm just curious,
like, do the rest of y'all, Was that kind of your reaction to it as well?
And I'm curious to just the conversations that you've had with clients like did
that did that stick with them at all about a way to, you know, think about both
like the combination of models they're working with and just like at all in
terms of the way that they're thinking about like infrastructure and compute.
Yeah, my reaction was the, I think it's the first architecture, which
takes security and confidentiality between a device I have in my pocket
and, you know, computation, which is on the other side of a wire, seriously.
And I think they really try to figure out a way, um, to keep like the, you know,
like they're selling a device and like everything happens on the device and they
figured out, well, we cannot do that.
And so they try to come up with an architecture where you, you, you
extend your trust domain into a data center and they, they went, I
think, Overboard with like, okay, we, you know, we have secure boot.
Okay. That's what everybody expects.
And then you have, um, non introspect ability of any, any data, which goes
in by the like by an administrator, so you don't have privilege elevation
of administrative accounts that I cannot extract data at all.
So I think there is.
They really, oh, and then we publish our binaries and the binaries are digitally
signed and they are only buildable through our internal build processes.
And so now anybody can inspect it.
So they really try to say, look what we guarantee you with physical isolation.
We are doing also in the cloud with like.
digital isolation, you know, through that, all these things.
I think the second really smart move was to say we are taking the same chip,
we are not buying, you know, an NVIDIA chip or, you know, an AMD chip, but we
are using our own infrastructure to run those things on our, on the same device,
like on the same operating system, right?
So they're M3 or whatever they have internally.
Um, and so, And now you can run the code on both ends.
So, like, from a development perspective, it really brings the cost down.
But if you look at those scales, those Apple scales, with, you know,
hundreds of millions of chips, like, it's really cheap for them, right?
So, a chip costs you, like, 60 bucks if you make it
yourself, put a bunch of memory.
The whole system is probably in the ballpark of, like, $200.
And they can stamp them out.
They know how to stamp out at scale, right?
And so, if you would do that alternatively with an NVIDIA card, Card and x86 servers
to cost would be like super ballooning.
And so I think there's a lot of really smart, but design decisions in there,
but they really looked at an end to end.
I think they figured out and they put the bar there, how consumer AI trustworthy
consumer AI will now look like.
And everybody will be like, okay, if you don't do it like Apple,
I will not use your source.
And I think now what's unclear is what the enterprise answer to that is, because
there are other questions which are asked.
But I don't think that's just in the consumer and inference.
So, uh, I run a Apple M3, right?
MacBook Pro.
It's a 128 gig unified memory.
Um, it's, this machine is a beast, right?
And basically, I can run Llama 3 70 B on my local machine...
because of the unified memory, I can say right now, there is no consumer machine
that I can afford, right, other than the Apple M3 that would be able to run
Llama 3 on on my machine, not only that, I can fine tune models just as fast as
I can on Google Colab on my MLX machine.
Without quantization, I can just take a base model and I can fine tune it.
So I fine tuned, I think it was 1, 000 rows, completely unquantized, so maybe
it was about 10, 000 tokens, maybe more.
Um, and I did that in less than 10 minutes on my, on my M3.
I didn't need to go to cloud, just did it on my machine.
Now, I haven't even measured that with quantization or Laura at that point, but
that is going to be a future for fine tuning as well, even in the enterprise.
Why would I go to a cloud to go and fine tune my data if I'm fine tuning maybe
100, 000 rows worth of data, right?
Maybe I'm doing a couple million tokens.
I can run that on MacBook, and half of my enterprise is sitting with MacBooks on
their machines, so I think it's, I think Apple's making a really smart play, and
I don't think it's just in the inference space, I think it's in the fine tuning
space as well, I think Apple has set themselves up as the real developer.
Uh, workstation for AI
I think combining this with something like what IBM is doing and Red Hat with
Instruct Lab would be really powerful because it also brings that kind of,
uh, creativity in terms of creating your own models, fine tuning them all in your
local space without having to deal with the complexity or the cost of the cloud.
I think that's going to be really powerful.
It's funny. I think one of the outcomes of just that whole thing was that from the
very start when people started to get educated about models, people were
like, fine tune as a last resort.
Like you should, you should do rag first.
You should do all these things before you ever think about fine tuning.
And I kind of feel like Apple made fine tuning cool because that was like such
an important part of what they're doing.
And it just, it does strike me as like, as it gets easier and cheaper
to do this stuff that we'll see, we'll see more and more fine tunes.
I certainly like hugging faces just like littered with fine tunes at
this point, but I think, um, it does feel to me like there's going to
be sort of more and more of that.
And I think that's actually sort of a great segue for the last segment
where we were going to just talk a little bit about, Model optimization.
There's a huge amount of activity in the space.
I think, um, you know, the count of and it even seems like the emphasis on
smaller models, even from the big players is getting to be more and more and more
like even some of the messaging I've seen from them recently is like we want
to deliver models that are actually like usable to the community as opposed to
just like this, just ultra AGI path, which is only useful in like some dimensions.
Um, you know, Chris, you just talked a little bit about Laura and quantization.
Um, I'm just, you know, beyond like, what are you all seeing in terms
of the things people are doing?
And that's having like the most impact in the market right now in terms of
Taking a combination of like small models and fine tuning all these techniques
and actually compressing them down to get their inference costs lower,
but still getting kind of the results that they're that they're looking for,
there's like a lot in this space, and there's a lot of energy in the space.
And I'm just curious in terms of like the things you've seen, the things you've
done, the clients you've worked with, um, you know, just where you're seeing
kind of the most bang for the buck and where people are kind of most excited,
um, right now and, you know, happy for, um, anyone to kind of take that one.
I think there's a lot of, you know, excitement around these model
optimizations, you know, reducing the complexity of the models, quantizing
the models, pruning the models, applying things like knowledge distillation.
So there is a suite of techniques right now and tons of great libraries out there.
Hugging Face is doing a, I think they're doing a fabulous job, uh, with their
Optimum, uh, uh, library, where they have multiple hardware vendors, and
they basically focusing on these, uh, hardware level optimizations, but kind
of abstracting them away where you can have these LLM extensions, for example,
for better transformers, for things like quantization with GPTGTQ and or for
bits and bytes, so these are libraries that make it very easy with some common
APIs to benefit from quantization, pruning, uh, better transformers or
optimization and transformer like this.
page attention.
Uh, you know, some of these quantization like you, Laura, et cetera.
So those are very important to kind of democratize and make it easy for
people to consume without really having to understand the depth and the
details of different hardware vendors.
So that's a good example of kind of democratizing this, all of
these optimizations and making them accessible to the developers.
Um, Of course, that requires a lot of work from different vendors to
be able to implement all of these optimizations and then provide
these common API for the end users.
But I see there's a lot of energy there and a lot of work and documentation
and, uh, which make it very easy, you know, for the end users and the
developers to use these techniques.
Yeah, and just to sort of add into that, I mean, there's obvious ones as
well, like kind of caching, for example.
One of the ones that I find quite interesting at the
moment is probably batching.
So there was a project somebody did with MLX earlier this week, where, you
know, if you use MLX out of the box, on my machine certainly, with something
like the Gemma, Model to be model.
You're going to get something like 100 tokens per second.
That's unquantized.
But they got up to just by batching the inference, they got it up to 1300
tokens per second on their machine.
And I've seen similar things with fine tuning.
So one of the things that I've done in my data pipeline is the MLX
pipeline wasn't fast enough for me.
So I rewrote all the Apple data pipeline so that it was just kind of
loading in and The biggest thing that I found is actually I pre process
all the tokenization for fine tuning.
So rather than letting it load, um, you know, as it's trying to fine
tune, I just pre tokenize everything.
I set up all the input and target tensors, get it all done in advance.
But the last thing that I do is then I bucket.
all of the batches together on the same sizes, which is kind
of almost like padding free.
But really what I'm doing in this case is bucketing based on the same
pad padding size to reduce padding.
And that is massively increased, uh, my speed of training.
So I think these techniques like caching training, you know, uh, cause I mentioned
things like you, Laura, et cetera.
Again, even, even, even if you think of the way LoRa works,
um, you're really looking at things like, um, freezing layers.
I think at some point, if I think of the Golden Gate stuff, the Anthropic we're
doing, I, I think you're going to get to the point, rather than doing a hard
freeze on layers, I think you're going to get, start to freeze on features,
because if you can understand Feature A and Feature B are being influenced,
uh, or is, are on a certain topic, why not freeze those features right in the
future and then train areas only in the feature area that you want to train.
So I can see that happening in the future as well.
So I think there's going to be a lot of optimized patients
coming in on those areas.
I know we're just about out of time.
So Volkmar, maybe I'll, as the first time guest, um, on the show, maybe I'll, I'll
give it over to you to maybe close us out and talk, you know, of all the sort of
like techniques that are in the market right now that are like really focused
on like optimization, which ones, which ones are you seeing, like, are Yeah.
Which ones are you most excited about and that you see the market
kind of most excited about?
So I think the similar to what Chris and Kaoutar already said, you know, I think
we get a bag of these optimizations.
What's interesting, uh, what I find interesting is the, uh, Uh, like
anything which is giving you like a 2, 3x speed up in whatever dimension.
Like we are not yet, I mean, when you compound them you may get to a 10x.
One of the things that really excites me, um, is speculative decoding.
So we are running a sim, like a simpler model, um, on top of your complex model.
And the simpler model is kind of trying to predict a bunch of tokens.
Um.
Ahead, and then you just verify it with the big model because all the
quantization methods, right, there's a quality impact and the quality impact,
you know, it's really hard to quantify.
And so we have these models which we're testing and then we go through
quantization and they kind of look good, but you don't know what you don't know.
And with speculation, you effectively verify against the full model and you
get just a 3x performance improvement.
And I think that's probably the one thing where, um, uh, you know, I,
I think that's, that's the biggest impact because it's also like batching
makes things better, et cetera.
And then you have like a continuous batching.
So, you know, requests come in and go out, but there's always, um, like
there's a latency impact with that one.
It's really like, it's just, it's better and you don't have
any, any reduction of quality.
I think that's a great place to end on.
And it's also just like thematically, I think one of the most fun parts about the
space, which is, it just feels like you can wake up one day and it's like, Oh, I,
someone struck together a couple two Xs.
And now we got a 10 X improvement.
Um, and something, you know, being at that stage of the journey is a lot
more exciting and interesting than, Some of the later ones, potentially.
So, um, it's a fun, fun time right now.
And, um, I think a great, a great place to end on.
So thank you all for, for joining us today for our listeners, for our guests.
And, uh, we will see you back next time on mixture of experts.
Thank you.