When to Use AI vs Agents
Key Points
- The video introduces a four‑category decision framework for choosing between plain data processing, classical predictive ML, generative AI, and AI agents, helping viewers know exactly when each approach is appropriate.
- Category 1 (plain data processing) covers simple cleaning, aggregation, and reporting tasks—any problem that can be expressed as a basic math formula should **not** use AI or agents because it’s slower, costlier, and less reliable.
- Category 2 (classical predictive machine learning) remains valuable for structured historical data with a clear target variable (e.g., demand forecasting, fraud detection, churn prediction), despite current hype around large language models.
- The later categories (generative AI/large language models and AI agents) are suited to tasks requiring natural‑language understanding, creative content generation, or autonomous workflow orchestration—situations where pattern‑based ML or simple queries fall short.
- The presenter also provides practical scripts and principles for pushing back on unrealistic AI requests from stakeholders, ensuring solutions are matched to the right technology category.
Sections
- When to Use (or Skip) AI - The speaker outlines a four‑category decision framework for data and insight problems, emphasizing that simple data processing tasks—like basic reporting or aggregation—should never involve AI, agents, or generative models.
- LLM Applications for Unstructured Tasks - The speaker explains how large language models are suited for generating textual (and image‑based) outputs from mixed, unstructured data such as summaries, drafts, and descriptions, while noting challenges like hallucinations, compute cost, and latency.
- Choosing the Right AI Ladder - The speaker stresses selecting the simplest effective AI solution—using a four‑rung ladder from basic data ops to AI agents—to avoid costly, overengineered implementations like using an AI agent for a simple sales sum.
- Cost‑Benefit Language for AI Projects - The speaker stresses the importance of framing AI initiatives in terms of data‑operations value and compute costs to persuade executives, highlighting generative AI’s expensive token usage, comparatively cheaper machine learning, and the relative ease of building and maintaining data pipelines.
- When to Choose AI Over Simpler Solutions - The speaker urges evaluating AI projects by first exploring non‑AI options and adopting agents or large‑language models only if they can deliver at least a tenfold improvement in accuracy, speed, or user experience, otherwise stick with simpler approaches and clearly articulate this rationale to leadership.
- When AI Really Replaces Humans - The speaker warns that many touted AI solutions still rely on humans, urging organizations to rigorously scope problems, benchmark transitions, and prioritize ROI‑driven value over feature lists to ensure genuine automation.
- Building Trust Through Tool Choice - The speaker stresses earning executive, personal, and customer trust by selecting appropriate technologies—agents, generative AI, data pipelines, and machine learning—and recognizing which problems each is best suited to solve.
Full Transcript
# When to Use AI vs Agents **Source:** [https://www.youtube.com/watch?v=1FKxyPAJ2Ok](https://www.youtube.com/watch?v=1FKxyPAJ2Ok) **Duration:** 00:22:01 ## Summary - The video introduces a four‑category decision framework for choosing between plain data processing, classical predictive ML, generative AI, and AI agents, helping viewers know exactly when each approach is appropriate. - Category 1 (plain data processing) covers simple cleaning, aggregation, and reporting tasks—any problem that can be expressed as a basic math formula should **not** use AI or agents because it’s slower, costlier, and less reliable. - Category 2 (classical predictive machine learning) remains valuable for structured historical data with a clear target variable (e.g., demand forecasting, fraud detection, churn prediction), despite current hype around large language models. - The later categories (generative AI/large language models and AI agents) are suited to tasks requiring natural‑language understanding, creative content generation, or autonomous workflow orchestration—situations where pattern‑based ML or simple queries fall short. - The presenter also provides practical scripts and principles for pushing back on unrealistic AI requests from stakeholders, ensuring solutions are matched to the right technology category. ## Sections - [00:00:00](https://www.youtube.com/watch?v=1FKxyPAJ2Ok&t=0s) **When to Use (or Skip) AI** - The speaker outlines a four‑category decision framework for data and insight problems, emphasizing that simple data processing tasks—like basic reporting or aggregation—should never involve AI, agents, or generative models. - [00:03:24](https://www.youtube.com/watch?v=1FKxyPAJ2Ok&t=204s) **LLM Applications for Unstructured Tasks** - The speaker explains how large language models are suited for generating textual (and image‑based) outputs from mixed, unstructured data such as summaries, drafts, and descriptions, while noting challenges like hallucinations, compute cost, and latency. - [00:06:49](https://www.youtube.com/watch?v=1FKxyPAJ2Ok&t=409s) **Choosing the Right AI Ladder** - The speaker stresses selecting the simplest effective AI solution—using a four‑rung ladder from basic data ops to AI agents—to avoid costly, overengineered implementations like using an AI agent for a simple sales sum. - [00:10:27](https://www.youtube.com/watch?v=1FKxyPAJ2Ok&t=627s) **Cost‑Benefit Language for AI Projects** - The speaker stresses the importance of framing AI initiatives in terms of data‑operations value and compute costs to persuade executives, highlighting generative AI’s expensive token usage, comparatively cheaper machine learning, and the relative ease of building and maintaining data pipelines. - [00:13:45](https://www.youtube.com/watch?v=1FKxyPAJ2Ok&t=825s) **When to Choose AI Over Simpler Solutions** - The speaker urges evaluating AI projects by first exploring non‑AI options and adopting agents or large‑language models only if they can deliver at least a tenfold improvement in accuracy, speed, or user experience, otherwise stick with simpler approaches and clearly articulate this rationale to leadership. - [00:18:08](https://www.youtube.com/watch?v=1FKxyPAJ2Ok&t=1088s) **When AI Really Replaces Humans** - The speaker warns that many touted AI solutions still rely on humans, urging organizations to rigorously scope problems, benchmark transitions, and prioritize ROI‑driven value over feature lists to ensure genuine automation. - [00:21:21](https://www.youtube.com/watch?v=1FKxyPAJ2Ok&t=1281s) **Building Trust Through Tool Choice** - The speaker stresses earning executive, personal, and customer trust by selecting appropriate technologies—agents, generative AI, data pipelines, and machine learning—and recognizing which problems each is best suited to solve. ## Full Transcript
Nate, when do I use AI? When do I use AI
agents? I get that question a lot. This
video is for you. If you've ever
wondered, how do I know when to use
agents? How do I know when to use
generative AI and large language models?
This is going to show you. We're going
to go through the four different
categories that you can choose between
when you make decisions about data and
insights. We're going to give you a
concrete decision framework. We're going
to give you the principles to work with
so you understand how to recognize these
problems elsewhere. And I'm even going
to give you scripts so that you can
understand if if an investor if your
boss is pushing on you for a solution
that you know won't work. How do you
push back in a way that makes sense?
First step, let's understand the four
categories we're working with. Number
one, plain old data processing. It's not
new. It's not fancy. It's not AI. It's
the simplest possible thing. Data
cleaning, aggregating it up, building
simple reports. If you just need to get
a very simple sales report and you're
aggregating up the clients and the
regions and over a particular time
period, that kind of thing, do not use
AI. Repeat after me, don't use AI. Don't
use agents. Don't use generative AI.
Don't believe anyone who tells you to do
that. If you're on an e-commerce site
and you just need to look at your
payment volumes over the last quarter,
don't use AI. If you just need to
understand how many SKs you have for
sale, don't use AI. Are you getting the
idea? If it is the kind of thing where
you could write it out as a math
problem, x + y= z, don't use ai. It's
not worth it. It's going to be much more
expensive. It's going to be less
dependable. It's a waste of everybody's
time. Let's go to bucket number two.
Classical predictive machine learning.
This one has almost disappeared because
there's been so much hype around large
language models. So, let's talk about
what it actually is and where to use it
so we don't lose the value. Because we
put decades of work into developing
classical machine learning. I've built
classical machine learning systems
myself at scale. It is important to
understand where it is still valuable
versus large language models, but almost
nobody thinks about it because of the
hype cycle, which is why we have to make
videos like this. If you have rich
historical data and you have a clear
target variable to optimize against and
you need something to predict that's
very specific, like I want to predict
seasonal Q4 demand or I want to detect
fraud or I want to predict churn. Well,
machine learning excels in taking
patterns in structured data and pulling
them to light when you have a clear goal
like that. Now, this takes training
data. It takes evaluation metrics. It
takes monitoring. It's a little bit more
complex than just running a SQL query.
But if you want to predict next quarter
sales based on past trends and
promotions, a lot of people are using
large language models for this. But the
correct tool is not large language
models. The correct tool is actually
traditional machine learning.
Traditional machine learning is designed
for situations where you have structured
data and a problem with a single
variable you're optimizing toward. Let
it do its job. Let it do its job. And
you see the difference, right? I want to
make sure you understand the difference.
If you're doing very simple sums and
reports and aggregations, that's not for
machine learning. That's that plain old
data processing category. If you want to
predict the performance of a single
variable and you have structured data,
that's not large language models. That's
not AI the way most people talk about
it. It's machine learning or traditional
artificial intelligence back before chat
GBT generative AI or large language
models. Now that's our third bucket.
Let's say you have a data set that's
mixed. You have some numbers. You have
structured data. You have some text. Now
you also need to generate text in the
answer. Maybe you need to generate
concrete summaries of the marketing
quarterly report and you want to
generate the text with that. Maybe you
want to generate an image with that. The
problem involves summarizing something
that is not numeric necessarily. It
involves translating things. It involves
drafting content. It has a lot of words
in it. Well, large language models are
probably your best tool at this point
for that. They're flexible, but you also
have to take into account
hallucinations, which I've talked about
a fair bit, how you handle higher
compute costs, and how you handle
latency, which is like the unpredictable
sort of gap in response time. So, if you
want to autodraft customer support
responses based on the text of the
customer support manual, that is a great
example of a large language model task.
If you want to autogenerate product
descriptions, that is a great example of
a large language model task. And you can
even do it if they're just looking at
the image and they're writing the
description based on the image. LLM
tasks are characterized by wordiness.
They're characterized by unstructured
data and they often have multi-threaded
output. So you're not optimizing for a
single variable in a structured data
set. You're actually trying to get
multiple outputs. You might have an
image output in some cases. You might
have a text output in other cases. And
you value that so highly that you are
willing to put up with the risk of
hallucinations and putting in guards to
minimize that and all the investment
that goes with that. In other words,
generative AI is more expensive to build
and maintain. So you have to do the math
to decide that it's worth it. And we'll
get into more of that ROI math later in
this video. The fourth bucket is AI
agents. It's the most complex bucket.
That's why I put it fourth. Use it when
tasks involve workflows. Dynamic
multi-step workflows with clear decision
points. That's critical for agents.
Decision points where you can describe
the criteria. You can describe the scope
of decision and you can give the agent
all the context it needs to make a good
choice. Things like scheduling,
follow-ups, data retrieval across
systems all fall into the agent bucket.
As an example, an agent that books
conference rooms, notifies attendees,
and adjust schedules when conflicts
arise automatically. That's an AI agent
problem. It's not traditional machine
learning. It's also not generative AI.
It's an agent problem. Agents can
orchestrate complicated tasks
autonomously, but you have to have very
careful error handling. You have to have
good observability so you can see what
they did and you need to have humans who
know how to debug them. So there's a
human talent question with agents as
well. It's worth thinking about though
because as we've gone through these four
buckets, what you should be thinking is
leverage. These buckets are not linear.
These buckets are disproportionate.
There's a power law return here. If you
get one x return on the simple x plus y
plus c, the simple monthly sales by
region, write a SQL query, you get it
back, you get x return on solving it
with a machine learning problem if it's
machine learning susceptible. You get a
100x return on generative AI and you can
get a x return on agents. Now those are
somewhat illustrative. I'm not saying
every single project falls exactly in
that number. But in my experience and
the experience of a lot of others who
have implemented these in practice, that
is how it works. These are not stairs.
It's like a roller coaster to heaven,
right? Like this is a crazy gain in
leverage as you move up, but it's also a
crazy gain in cost and maintenance. And
you have to design the more advanced
systems very intelligently and target
them at the right problem, which is
exactly why we have this video. Because
you can imagine the expense, the cost of
building a generative AI system, of
building an agentic system against a
problem set that didn't need it. What if
you built an AI agent workflow to sum
monthly sales by region? Is it possible?
Yes, absolutely. Is it like bringing a
bazooka to kill a fly? Yes, it is. It's
ridiculously expensive. You don't need
to do it and it would be a terrible
waste to try and do it that way. And
that brings me to the idea of the
ladder. Pick the simplest solution on
the ladder. Imagine these four rungs.
You have just basic data operations. You
have machine learning traditional. You
have generative AI and then finally you
have AI agents. Pick the lowest rung on
the ladder you possibly can. And I'm
going to show you how you kind of think
that through. You could call this
developing engineering taste, but it's
not just for engineers. So many of these
skills have been sequestered away and
hidden in engineering uh conference
rooms for too long. And I want to bring
them out because they're not actually
too technical and we really need them in
the age of AI. So the first skill that
you need to navigate this ladder
correctly is to focus on pattern
recognition over hype. Did you notice
how I talk about the type of pattern?
That is a skill you can learn to focus
on problem structure, not on buzzwords.
You can learn to ask, hey, what needs
solving? Is this a decision that we need
to solve for? Is it a prediction we need
to solve for? Is it a generation we need
to solve for? If it's a decision in a
workflow, it might be an agent. If it's
a prediction, well, that might be
traditional machine learning. If it's a
generation problem, that might be a
large language model problem. If it's
just a report, that might be traditional
just data operations. That kind of
thinking, that kind of sober focus on
problem structure is going to help you
resist the AI for everything temptation.
Sometimes just having a SQL uh script
that runs will solve almost your entire
problem. And by the way, I said almost
on purpose. Do not give in to the
temptation to make something a
generative AI project or an agent
project if 5 or 10% of the value is
coming from that agentic piece or that
generative AI piece and most of the
value is coming from SQL. If your boss
says to you, I want a quarterly
marketing report and I want it to have
this fancy insight as to why we why why
we performed the way we did and I want
it to be in text and I want to have an
illustration of our top selling product.
You could look at that and say, "Well,
there's some stuff here that is
generative AI." So, it's probably a
generative AI problem or maybe it's a
mixed problem where you use two or three
runs on that ladder. I talked about data
processing, generative AI, maybe even
some prediction from machine learning. I
would not look at it that way. Instead,
I would look at it and say, why the heck
do we need the picture? Why do we need
the text? Don't we get the business
value to make good decisions out of
traditional data operations? If you get
90% of the value for 5% of the cost, the
business should take that trade all day
and you should be able to articulate
that in dollars and cents very very
clearly. It is worth asking where the
leverage and the problem lies. That is
my point. So think of the problem as a
distribution and it's going to be
distributed along the four legs of that
ladder. If the problem is is going to be
bumpy and like really skewed heavily
toward one of the legs of the ladder,
you should take that pretty seriously.
You should say maybe this is
fundamentally just a data operations
problem because most of the problem
value is there and maybe we should make
the decision to cut the five or 10% of
value you're talking about and make that
a later choice and just do the thing
that we can get away with now that is
only one of the legs of that ladder. The
simplest one you need to understand how
to speak the language of costbenefit to
make these kinds of claims because
that's how executives speak. If your
boss or your investor is telling you to
invest in AI, the only way they will
really hear you is if you come back with
cost benefit. So start to learn how to
talk about compute costs. Generative AI
and agents are both extremely expensive
in tokens. It is not cheap to run those
pipelines. If you make a mistake with
that architecture, you you can be out
thousands, tens of thousands more.
Machine learning is a lot cheaper, but
does take expertise to set up and it's
not free. And data processing is the
cheapest of all. Almost anybody at this
point with an engineering degree can set
up a data processing pipeline without
any kind of issue. Most of us who don't
have engineering degrees can figure it
out with Jad GPT. The maintenance
version is also non-trivial and I want
to call that out because the maintenance
version you know how I talked about this
idea of power law returns and this
roller coaster that stretches up as a
way of showing like how much uh well
potentially fun but also how much
excitement uh there is in some of the
agentic use flow cases, the generative
AI cases. The maintenance also scales
with that. So if you have an agentic
workflow that is going to have
significantly more costs than a large
language model workflow that is going to
have significantly more costs again than
a machine learning workflow which is
still more expensive than a data
pipeline workflow. It's not 1 2 3 4
costs. It's more like 1 2 48 costs.
These costs get much more expensive. And
part of why is that agents and LLM
systems have to be maintained in
production. Your leader who charges you
with building these systems has to know
that the dollars keep going out the door
on time spent supporting these systems
after they are launched. And so instead
of thinking about an agent workflow like
traditional software, you have to think
about it as continually maintained
almost like a little employee that you
have to pay every month. The last thing
I want to call out from a costbenefit
perspective is time to value. As you
would imagine with compute costs, with
maintenance burden, it is increasingly
complex as you move up the ladder and
that takes increasing time and talent.
And so if just about anybody can set up
a data pipeline in a few days given the
data and if you can get a data scientist
to work with you on a machine learning
model and that might take just a couple
weeks if you have everything ready.
Generative AI prototypes really vary. If
it's out of box and it's super simple,
it can be a couple hours. And if it's a
full production pipeline, it's going to
be multiple weeks, multiple weeks. It's
not going to be easy. Often months, I
know people who are at scale, who are
building LLM pipelines that aren't
agentic, still taking them months. This
is not easy to do. Agents, if you're
starting from scratch and you're a
scrappy startup and you're wellunded in
the valley, sure. Can you knock up some
agents over the weekend? Absolutely. You
have the right talent. you have a clean
slate to work with. If you're an
existing company and you're trying to
get this done, it is even harder than
LLMs. It is quite difficult to do well
and quite difficult to sustain well and
you have to recruit the talent for it.
The months stretch out into 6 months or
more very, very quickly. And it's your
job if you are given this assignment or
asked to do a project that involves AI
to find a way to articulate that and
explain it to find a way to say yeah we
could do this with agents but it seems
like what you're really optimizing for
is just getting the data into a report
and there could be a simpler way to get
you that value much much faster. we can
find another use for AI so that you can
tell the board right so in sum if you
have someone coming to you or if you're
asking the question when do I use AI
when do I use agents before proposing AI
identify the simplest nonAI solution
evaluate whether AI measurably improves
the accuracy speed or user experience
against that solution and by measurably
I mean it has to significantly improve
my rule of thumb with using a large
language model or an agentic workflow is
that if it isn't 10x versus the
baseline, it probably isn't worth it
because there's adoption, there's
talent, there's systems to maintain. And
so 10x is my rule of thumb. And if you
don't hit that, stick with a simpler
approach. It's worth it. So I want to
suggest to you that there are a few
scripts that you can use if you get
stuck with leaders who are just not
believing you that will help you to work
through this. And then we're going to
get at the end of the video into some
sort of contrarian insights, things that
kind of go deeper. But before we do,
just to like sum up this piece around
costbenefit and communication. Here's
how you answer. Let's say your VP says,
"We need to use chat GPT for our
report." Common. I've seen it happen.
You could say this. Hey, uh, thanks for
asking. I researched three different
approaches for uh, the report problem
with a data pipeline. Uh, it's going to
take us 2 days to set it up. Uh, it's
going to cost us like 200 bucks and
we're going to have 100% accuracy on our
known metrics. You can have it by
Friday. If we used a machine learning
model, which is not AI, Mr. VP. That's
going to be 2 and 1/2 weeks with our
data science team, push back another
project. I suspect we'll get to 80%
accuracy on prediction initially and
have to work up from there. Uh total
cost, I want to say $15,000 maybe. If
it's generative AI and uh it's a large
language model prototype, I would guess
out of the gate that you're going to
have inaccuracies across all of your key
metrics initially. It will take us a few
days to get it set up and it will take
us probably three or four months to root
out enough of the hallucinations to make
the report really worthwhile. I would
recommend option one because we can get
reliable results by Friday and it's the
cheapest overall. If we really need
predictive insights, we can add that
machine learning model as a fast follow.
You see how that's something that speaks
executive. It talks cost. It talks time.
It doesn't say no directly. It actually
just reframes the problem and helps them
understand what's going on. Okay, now
let's get to some of the contrarian
insights that you need to have in your
head when you're facing these when do I
use agents, when do I use AI problems.
Number one, data quality is going to
beat model complexity every time.
Garbage in, garbage out, right? If you
are introducing AI and you have bad
data, you are pouring money down the
drain and lighting it on fire. Fix your
data pipelines before reaching for
models. And if you have great quality
data, you can use cheaper and cheaper
and cheaper models. Make sure you take
advantage of your data quality. And if
you don't have data quality, make sure
you make that a priority and fix it. You
do get real leverage from that fix. That
is how you unlock the 100x,000x use
cases cuz agents also struggle with it
if the data is bad quality. Number two,
boring solutions when a clear,
well-designed BI dashboard that just
works outperforms a fancy AI model that
nobody understands and that can't be
audited. I love AI. AI has a lot of use
cases, but I have seen too often now
that we're in this sort of hype cycle
for AI, people are throwing out the
boring solutions that work. Don't do
that. Don't be that person. Find ways to
reframe like the sort of narrative that
we talked about. Number three, human in
the loop first. When you're deploying
LLM, when you're deploying agents, start
by surfacing suggestions for humans to
vet, build trust, gather feedback, and
then automate. Now, if you're a startup
and you have nothing to lose, sure, give
it a shot. See if you can automate it
and make it work in a weekend. If you're
a company with stakes, you have to take
seriously the idea that humans need to
be involved from the beginning and
helping to get the model to a place
where it works. And you need to assume
that cost. Who's going to staff for
that? Who's going to pay for that? How
long are you going to use the humans?
How do you know if the humans are able
to transition to AI systems? You know,
one of the dirty secrets in AI is that
sometimes it never makes it and then
there's a scandal. So, for example, in
the Amazon just walk out checkout
stores, the company would trumpet that
it was AI that they were using, but the
reality was it was people looking
through the cameras, checking all the
work because they were never able to
transition out of humans in the loop. AI
is hard. Don't believe every pressed
release you see. Look at whether you can
reasonably hand off human work to AI.
Sometimes you can. There are real wins
out there, but think about it and make
an intentional plan and benchmark
yourself and see if you're actually able
to successfully do it. This comes down
to scoping the problem very very
precisely around the value you intend to
deliver. Number four, ROI focused. Don't
think about the feature list. Remember
when I said earlier in this video that
you want to mentally assess the problem
space and look at where the value is
spiky against that ladder, whether
that's simple data operations or machine
learning or generative AI or agents.
Focus on that value piece. The feature
lists will spread out all over the
place. If you focus on feature lists,
you're going to be extremely
inefficient. Focus on where the leverage
is. Focus on the value and frame AI as a
means to specific business outcomes. It
should deliver reduced costs, faster
decisions. It should deliver better
customer satisfaction. The point is not
AI itself. The point is the value it
delivers. And so if you need to push
back when someone makes the case that
you should use an AI agent and you have
used this framework and you know you
shouldn't, push back on ROI. push back
and say, I want to deliver reduced costs
and this isn't going to do that. I want
to make sure that we make good decisions
to actually improve customer
satisfaction, and this agent workflow is
not going to do that on the timeline you
need. So, let's close with a 30-se
secondond decision tree that will help
you the next time you face this. Does
the problem have deterministic rules?
It's a data processing problem. Does the
problem need to predict an outcome? It's
a traditional machine learning problem.
Does the problem need to generate novel
tokens, novel words, novel content,
generative AI, large language models?
Finally, do we need a workflow with
autonomous decisions and multi-step
orchestration? That's an agent's
problem. The cross cutting factor here
is talent. As you go up these four, the
talent needs get bigger. The same person
that can do data processing usually
can't do autonomous multi-step
orchestration with agents, not a
production scale. So you have to have to
also be aware of how to advocate for the
talent that you need in order to deliver
against these outcomes. It is not fair
to ask an engineer who has never worked
with large language models to
immediately build an autonomous
multi-step orchestration agent that
handles 95% of customer service tickets.
That is unlikely to go well and the
companies that tend to ask for that are
setting themselves up for grief. I hope
this has been helpful. My goal for you
has been to help you to develop a sense
of taste by focusing on problem
structure, not problem hype and not
solution hype in AI. To start simple,
remember to climb the complexity ladder
as you go. To make sure you frame things
in terms of ROI, that's going to help
executives to really make sense of what
you're saying. To build trust first,
which by the way is one of the
underlying themes all the way through.
You have to build trust with your
executive by speaking their language.
You have to build trust with yourself
when you're making your decisions by
making decisions that actually lead to
sustainable software. You have to build
trust with your customers by making sure
that you focus on quality for them and
use the right tool. Data pipelines,
machine learning, generative AI, and
agents each have their place. You need
to know when to pick up the right tool
in the toolbox. I hope that this video
has given you a sense of the kinds of
problems that are susceptible to agents,
the kinds of problems that are
susceptible to generative AI, and the
frankly fairly wide class of problems
that isn't either of those things and
that we still have in business today and
that still needs a good solution. Good
luck out there.