The True Cost of Generative AI
Sections
- Examining the True Cost of AI - Two industry experts debate whether AI is truly free, exploring its hidden financial impacts on cloud adoption and enterprise modernization.
- Hidden Costs of Scaling AI - The discussion reveals overlooked expenses—including data acquisition, governance, and integrating model inference results into backend systems—that become significant as AI initiatives expand beyond pilot phases.
- Navigating Cloud Transformation Expenses - The speakers explain how companies must invest in automation and foundational cloud projects—often over a two‑to‑four‑year timeline—while realistically addressing both upfront and hidden costs and offering strategies for firms that lack the necessary budget.
- From PoC to Production: AI Governance - The speaker explains that scaling AI pilots into production faces regulatory, security, and governance hurdles, requiring a dedicated AI Center of Excellence to manage risk and efficiency.
- Assessing Generative AI Adoption - A discussion on how companies can evaluate the need for generative AI by weighing infrastructure requirements, business involvement, competitive pressures, and beginning with efficiency‑focused pilot projects.
- Future‑Proofing Investments with Flexible Architecture - The speakers emphasize that designing adaptable, multi‑cloud and vendor‑agnostic solutions safeguards early projects from becoming obsolete as technologies shift, turning even discontinued efforts into valuable learning for future directions.
- AI Investment: 50% Gains and Competitive Risk - The speaker argues that adopting AI can deliver roughly 50% efficiency improvements and revenue boosts, but warns that failing to invest risks losing competitive advantage and slower time‑to‑market.
Full Transcript
# The True Cost of Generative AI **Source:** [https://www.youtube.com/watch?v=xerfuLpaApk](https://www.youtube.com/watch?v=xerfuLpaApk) **Duration:** 00:21:14 ## Sections - [00:00:00](https://www.youtube.com/watch?v=xerfuLpaApk&t=0s) **Examining the True Cost of AI** - Two industry experts debate whether AI is truly free, exploring its hidden financial impacts on cloud adoption and enterprise modernization. - [00:03:09](https://www.youtube.com/watch?v=xerfuLpaApk&t=189s) **Hidden Costs of Scaling AI** - The discussion reveals overlooked expenses—including data acquisition, governance, and integrating model inference results into backend systems—that become significant as AI initiatives expand beyond pilot phases. - [00:06:19](https://www.youtube.com/watch?v=xerfuLpaApk&t=379s) **Navigating Cloud Transformation Expenses** - The speakers explain how companies must invest in automation and foundational cloud projects—often over a two‑to‑four‑year timeline—while realistically addressing both upfront and hidden costs and offering strategies for firms that lack the necessary budget. - [00:09:23](https://www.youtube.com/watch?v=xerfuLpaApk&t=563s) **From PoC to Production: AI Governance** - The speaker explains that scaling AI pilots into production faces regulatory, security, and governance hurdles, requiring a dedicated AI Center of Excellence to manage risk and efficiency. - [00:12:30](https://www.youtube.com/watch?v=xerfuLpaApk&t=750s) **Assessing Generative AI Adoption** - A discussion on how companies can evaluate the need for generative AI by weighing infrastructure requirements, business involvement, competitive pressures, and beginning with efficiency‑focused pilot projects. - [00:15:35](https://www.youtube.com/watch?v=xerfuLpaApk&t=935s) **Future‑Proofing Investments with Flexible Architecture** - The speakers emphasize that designing adaptable, multi‑cloud and vendor‑agnostic solutions safeguards early projects from becoming obsolete as technologies shift, turning even discontinued efforts into valuable learning for future directions. - [00:18:43](https://www.youtube.com/watch?v=xerfuLpaApk&t=1123s) **AI Investment: 50% Gains and Competitive Risk** - The speaker argues that adopting AI can deliver roughly 50% efficiency improvements and revenue boosts, but warns that failing to invest risks losing competitive advantage and slower time‑to‑market. ## Full Transcript
A lot of Gen AI says it's free.
But is anything in life really free?
So with all of the hype around using AI everywhere, it's worth it to ask
should we be using AI everywhere and can we afford it?
So I wanted to ask two people who are on the front lines of this question.
Rebecca Gott and Penny Madsen.
Welcome, Rebecca and Penny.
Glad to be here.
Thanks for having me.
Great to see you.
Now, Penny, I want to start with you, though, first.
Can you tell me a little bit about how you got into talking about the cost of AI?
Was quite a long road.
I started off as a journalist covering technology, and back then, really,
all the conversations were around
ERP and CRM and and very little about data centers,
and then ended up focusing on data centers and saw the rise of that industry.
And I'm now at IDC, where my focus is looking at
what's happening with the cloud buyer, what challenges they might have, what's,
and what's making them adopt different models of cloud,
and what external influences from macroeconomic influences
to things like gen AI are having on the industry.
Oh, wonderful. You've got such a diverse background.
I can't wait for us to like, pick into that today.
Now, Rebecca, what about you?
So I'm CTO for IBM Power Platform, and so I spend my days
working with customers, understanding where they need to take their businesses
and where they need to take their IT landscapes.
And much of that discussion
has been around modernization, essentially taking their,
some of their their existing applications, their workloads,
and taking them to essentially the next level, modernizing them.
And that's where often AI enters the picture.
So then I know that
they always have questions like, Rebecca, do I really need this?
Exactly.
Yes. Yeah.
And as you know, there's so much talk within the industry
and just the general public around AI and the uses use cases around AI.
So businesses are no different. Great.
Well we're going to get into all of that.
Penny, like I know that you also are somebody who has a ton of conversations
with people who've started big AI initiatives.
Point blank. Why is it so expensive?
There's a multitude of reasons why this is so expensive,
and I don't think it's even as expensive as what it's
going to be for a lot of companies going forward as well.
We sort of see that at the moment, about 1.9 million
per company for a big company is going forward in 2024,
but this increases to 4.1 when we look at what they expect
to spend in 2025, and I think you've got to look at it from
both the infrastructure viewpoint, what's happening with the applications
also the amount of cultural change and business change
that has to take place in the industry.
Things like skills, for example, which are highly critical
and increasingly expensive, the network, the infrastructure.
When you're looking at having higher performance compute
and then all the business change, and then you've got to think all of this
is all based on data and and just the cost.
And being able to work out the data governance,
the right storage requirement and get this all into play.
It's not cheap.
A lot of companies are only doing this in very small scale at the moment.
But I think as we start to scale out, we're going to see costs
look very different.
So as you continue to reveal these costs to me, Rebecca, I know that
you probably got some good insight when it comes to some hidden costs,
some things that maybe are laying
within some of those layers that Penny just broke down for us.
Can you please, like, pull the curtain back even more for us?
Absolutely. Yeah.
So, yeah.
So so one thing Penny hit on was just the data.
And often, you know,
when we're working with customers and they have ideas around use cases,
one of the first realizations can be, do we even have the data?
Do we collect the right data?
And that can actually take some amount of time, sometimes quite a long time,
to put the right mechanisms in place to actually collect that data.
You also have to think about the data governance.
You know, basically
what are the controls, you know, the privacy permissions around that data.
The second thing that we often see is and so then you have much further
and I'm going to skip a few steps here.
Another I'll say kind of a hidden cost that isn't necessarily thought of...
thought about up front
is the integration of those results into the back end systems.
Often the,
you know, the building the model, the you know,
you have the model in place, you're doing inferencing.
But once you start getting those inferencing results,
how do you integrate that into whatever the back end data system is?
And that can take some amount of time.
And so one example I'll point to is
we are working with a hospital in Asia Pacific doing pathology. And
their inferencing was on essentially
digital probes and it was for essentially object
recognition image classification type of thing.
And so they had the inferencing in place.
But then to take those results and actually integrate them
into their back end system of pathology information system.
That actually was another journey for them to actually make use of that, that data.
So that integration piece can be a big piece as well.
The other thing that we see, which I, I've been doing a lot of work
on recently, particularly
because I'm looking at what's happening with cloud bias, is a lot of organizations
might be starting on this and they go, we want a gen AI project.
They then quickly find out that they've got a data governance project.
So they've got to go back to square one with data and interestingly,
we see, for example, companies in Western Europe might be a little bit
more prepared for some of this because they've dealt with data
sovereignty needs with GDPR and other things.
They've looked at their data, but a lot of companies
then have to get their data into the right shape to do this.
And then they need to also factor in the account
that actually can our infrastructure enable us to do this.
And what we're finding is a lot of companies
are then finding out that they don't have the right cloud foundations.
They haven't set themselves up for the hybrid cloud.
Companies want to access a lot of data from where it might be resting,
or they might need to get data to somewhere else,
or use different models, and they might find that that then
with the exorbitant network costs.
So they need to have the right automation and processes in place.
So companies are then having to go back and carry out
a cloud transformation project to build the foundations for this.
And we find three quarters of companies are actually on that journey today.
And they want to have this done in two years, three years, four years.
They're not thinking five years.
And it's pretty transformational.
They're looking at changing up
to three quarters of any cloud strategy they might have in place already.
Okay.
First off, I want to take a moment to thank both of you for keeping it 100%
real.
Neither of you are painting a fantasy of like, here's how easy it is.
Here's how you can get AI on the cheap.
That's not what this podcast is, right now, so thank you for keeping it real
there.
We've talked a bit about some of these hidden costs and back end costs.
And you know, Penny, as you mentioned,
you enter the project like with your eyes focused on one specific area.
And then you realize, oh, hold on. This is actually a data management thing.
So I want to talk a little bit more.
I want to go in reverse and talk about some of the upfront costs.
Right.
So before we even get to some of the hidden ones,
what if a company is not prepared for the full upfront costs?
Like is there any space that they can move
into next, or some way that they can help to place them in that position?
Like, what can you do if you don't think that you can afford it?
We have an IBM Institute of Business Values that does surveys occasionally,
and we did and we did one recent survey, 47% responded that
they don't have a handle on what to expect for the cost of AI.
How much will it cost?
They're seeking essentially advice through external services, third parties
that have been through these processes with other customers
to get a scope on the upfront costs, and then they can step back
and work on digesting what what they can potentially bite off.
We actually find that there's a pretty high number of projects
that we've been tracking that don't end up realizing the true value
or return on investment that companies want to get from them.
It's early days, and that's where we're finding
most of these projects are being that linked to business KPIs.
They're having to look at
driving consolidation and cutting tech debt wherever they are.
And they're already on this journey
because of inflationary pressures to make things work better and faster.
But now they've got this added pressure of doing it for gen AI, which requires
so much more.
There's a lot of organizations are just out there playing around
on the licensed models, for example, that might seem cheap to start with,
but then, you know, very quickly when you look to scale those,
they can ramp up in costs and you might find other problems as well.
So I think it's really,
there's a difference in companies doing the proof of concepts
to getting those into production.
And that's where the cost comes in.
It's one thing to step into a POC and POCs actually can be really great
learning vehicles for companies, you know,
particularly as they're entering the space to get a handle on
a true understanding of the skills that are needed,
the type of infrastructure that may be needed.
And so getting through that POC, actually there can be tremendous,
really solid learning.
But then taking that next step in terms of okay, we have this great
and sometimes successful POC, but taking it into a production
setting is a whole other barrier of entry often.
And so that's when you really start to need to understand
any, you know, regulatory processes or compliance issues.
Know data governance that you need to be mindful of and security also.
So security often can be an afterthought.
So that barrier to production, it can be it can be a big one.
I was talking to a company recently.
I was doing a number of interviews with organizations that have been doing
AI projects, and it was really interesting because so many of them said
we just had this directive from the C level really quickly through the I.T.
team, go get some gen AI projects on the table.
We want to see things working
and they, in a lot of cases it was put out to the whole company.
What can you do with gen AI, go do what you can.
We've opened up some licenses and nothing happens.
Nothing happened until organizations actually formed their own center
of excellence around AI, or had a team that can actually manage this.
Because could you imagine
if you had everybody in the company doing their own gen AI projects?
One, lack of efficiency across the whole business,
but then lack of the ability to then be able to look at
and orchestrate these projects, and then opening up the element of risk
as well that might come from doing some of these.
Penny, you said three words that made my ears perk up.
You said a lot of words,
but three in particular.
Now that I want to drill in a little bit more on, Center of Excellence,
that sounds like a really cool academy, but I'm sure that that's not exactly what
that is in the context of what you shared.
Can you tell me more about exactly how a center of excellence works?
Yeah.
So I actually see this is this is the exact same thing, right?
In terms of exactly what Penny is saying
is that we work with companies and it's so spot on.
They, there's a directive,
you know, for to think of do ideation, they think of all these great ideas.
And so we talked to some you know some teams and different lines of businesses
are coming in with really great, you know some really great ideas.
But there has to be some also some sort of, I'll say kind of center
to guide and lead the efforts.
You see, chief data and analytics officers being named or even chief
AI officers being named to put some organization around it
and establish best practices, you know, with regard
to data governance and tools and processes to get a handle on things.
Because what you want to prevent is, is kind of exactly what Penny was touching on
was that this kind of risk of, I'll say, shadow I.T.
where you have the different departments, you know, that everyone have different
department tends to have their own funding lines,
so you don't want them buying their own infrastructure, kicking off,
you know, buying their own licenses and having this very, I'll say,
very different ways of doing things just depending on what,
what team you're working in.
And so so I think that's actually
a really good practice that companies are understanding.
They need to establish, essentially that center of excellence.
Yeah.
And I think the lines of communication as well, being able to have
that coordination across line of business and cloud,
because these gen AI projects, they're not just about gen AI.
There is that underlying infrastructure element here.
And considering the impact some of these will have on that.
But also,
you know what the infrastructure needs to do to be able to accommodate this.
So we're actually finding more and more that we're getting lines
of business become involved in cloud decisions, for example, that,
at the
preliminary stage and test and development, for example, where
decisions can be made and should be made about where things are housed and located.
Can either of you, or both of you together,
can you help me get some best practices in terms of how can a company
really drill down and figure out, do we need generative AI?
Because you both have done a beautiful job
of painting a lot of the barriers and a lot of the concerns,
but how can one know that this is something that's still worth it
for us to enter into?
So I think one thought is, would generative
AI, introducing it, either embedding it in your products
or introducing it into your back end processes for developer
efficiency, let's say, does that give you a competitive advantage,
or do you believe your competitors are pursuing these technologies?
And will
that give them a competitive advantage that then basically you are left behind?
Yeah, and I think organizations that are doing this really well are focusing first
on those projects, for example, that might drive some efficiency
and bring them pretty direct small gains and enable them
to actually get their head around what's happening here.
But I think also a lot of companies are looking at gen AI and saying, do
we actually need an AI?
Maybe we just need AI.
We’ve gotta remember, there are other tools around before Gen AI.
If I'm an organization that's going to take your advice, Penny,
and think about perhaps considering do I need
gen AI right now or is AI enough?
Are there resources that have not even been tapped into yet
that I could be tapping on into?
It seems like that's one way that as an organization,
I could help to keep my costs under control.
Can both of you kind of give me some more ideas in terms of other ways
that companies might be able to keep their costs under control if they know?
Like you said, Rebecca, hey, my competitor is over here already leaning on in.
I can't afford to not invest, but how can I still do this in a considerate manner?
Taking a look at essentially, I think starting small is a good approach.
What we're seeing is more customers are understanding that AI really
is a fit for purpose with regard to infrastructure.
And so what can you do with your existing I.T.,
assets, you know, can you start with essentially,
maybe the compute farm, the servers you have in place
or do you need some, you know, very expensive GPUs?
Often customers are finding they can start smaller and then learn
and then see where they need to go to scale.
Yeah, I think also anybody going down this route needs to think very carefully
about what their partners are doing down this route.
You need to consider who you're working
with, what flexibility, because there's going to be so many innovations
take place in this space.
What you're using today,
there might be something different you require in a year's time.
So having that flexibility built in.
But flexibility isn't just about can I access different clouds?
Can I access different environments?
Can I do things at the edge?
It's not just a technical question.
It's also about the contracts and the arrangements you might have.
It's about where the vendors that you're working with on this are going,
because no one vendor is going to be able to meet every need for every organization
in this space.
Well, then how often have either of you seen companies right in the midst
of making that initial investment, and then, as you mentioned, Penny,
there's some sort of industry shift or a technological leap
that then kind of makes that initial investment
or that initial project that was being worked on kind of obsolete?
If you embark on a project and it ends up being valuable,
but you don't want to take it
further, there's generally still some very good learning behind that.
That informs essentially where you want to go next.
And so, so I tend not to think that things are just throw away.
And it was, you know, I'll say a wasted effort
because there is learning and understanding of,
okay, we're going in this direction, but maybe we need to shift a little bit
and go in this direction because, hey, there's this new tool or technology.
And as Penny mentioned, the whole ISV and the ecosystem,
there's so much new things and services coming out, you know, enabled
through our system integrators, you know, the partners you work with
and just having that flexibility, companies need to have that in mind.
So I guess the one constant
for a lot of organizations is that nothing is constant anymore.
We don't know what's going to be around the corner.
And gen AI kind of brings us to the, it’s the
technical version
of that, I guess, versus adding to those pressures.
But we can't forget that now, there's other macroeconomic things
happening behind the scenes.
There's a certain political element to some of this.
There's definitely,
also a regulatory element that we don't know what's going to change there.
So once again, it's a moving space.
And to protect themselves,
businesses really should set themselves up so they can think on their feet.
So if I'm a business that's listening right now, then I've heard
both Rebecca and Penny, I’ve heard both of you basically say, though,
that AI is absolutely something that can help businesses to compete.
You've just got to figure out
what is the right AI or gen AI approach for yourselves.
But if I'm a business that's now discovered, okay, so I need to do this,
what would you tell me in order to get me to actually foot the bill?
Because I've still now got to write that check in order to make the investment.
What would you say to help to put me at ease and or to inspire me?
At a very minimum, understanding what the technology can do for you.
Start that education.
That can be some initial investment in terms of basically just just learning
and understanding possible use cases
that have a return on investment that you find attractive.
And so I think looking at some of that as very, I’ll say,
just opening steps can then lead to the next step.
To put it in context,
if I had to say to somebody, you need to be investing in this today,
a lot of the proof of concepts
that I've seen become production successfully.
The returns I've seen or heard about
have all been around that 50% figure.
So companies
saying we're seeing 50% better efficiencies, they're seeing 50% savings.
They're getting 50% more back methods for putting in in terms of revenue.
So there's definite advantages there if you can get it right.
But I think, think about the foundations because
even if you do one project, you want to be able to replicate that
or you want to be able to scale that project,
you want to be able to turn it around and do things quickly.
Take that knowledge that you've built up,
and the only way you're going to get there is by investing.
Okay, I like that and I like that.
That 50%, number too, that it's quite compelling.
Now let's look at the other side of the coin.
What am I risking if I don't take the leap.
What are companies
potentially going to miss out on if they don't take the leap into investing in AI?
I think, competitive advantage, and we do see many, many companies
looking at some of their back end processes and efficiencies.
So what can I do to enable gen AI or just
AI tools that add to my developer efficiencies?
And that can really pay dividends.
If you are able to get your products to market faster than competitors,
because you have put in efficiencies into your back end development processes,
that actually can be a winning game and lead to true competitive advantage.
So I also think there's going to be you mentioned skills.
And the reason why skills are so expensive is it's really competitive
for those skills at the moment.
And if you're not doing something around this,
the next generation of people coming out looking at whether they're going to be
working are going to be wondering why, and it's going...
you're going to find yourself becoming more and more of a laggard
with some of these industry adoption areas.
Well, look, I am, I'm very inspired right now.
And on behalf of our listeners, I'm just going to assume
that they're also really inspired because they've listened this long.
So thank you all for listening.
but Rebecca Pennie, thank you both for being here today,
for sharing insights, for being so generous with this discussion.
I really appreciate it.
And everyone who is listening, everyone who's watching.
Thank you as well.
If you happen to have any additional thoughts about this episode, please
leave them in the comments and we will see you next time.
Promise.