Orchestrating Enterprise Data and AI
Key Points
- Successful enterprise AI projects are likened to a symphony, where technology tools act as instruments that must be coordinated and guided by a clear “sheet music” (strategy and processes).
- Choosing the right infrastructure (on‑prem, cloud, or hybrid) and optimizing it for storage versus compute depends on the specific data types and use‑case requirements.
- Data originates from many sources—point‑of‑sale systems, CRM, finance, etc.—and must be integrated across legacy IT environments and newer cloud‑native platforms to support modern analytics.
- While rapidly evolving AI/ML tools and large language models generate valuable insights, domain‑savvy business stakeholders are essential to frame the right questions and turn those insights into actionable outcomes.
Sections
- Untitled Section
- Collaborative Data Insight Process - It emphasizes that business users, IT, and data scientists must work together to harness rapidly evolving technology tools and diverse data sources—collecting, integrating, and analyzing the data—to answer key business questions and create actionable insights.
- Integrating Lakehouse and Data Fabric - The speaker explains how combining a data lakehouse with a data fabric architecture, alongside proper processes and user involvement, is essential for organizing and extracting value from enterprise data across diverse environments.
Full Transcript
# Orchestrating Enterprise Data and AI **Source:** [https://www.youtube.com/watch?v=i0HDXfCXPLA](https://www.youtube.com/watch?v=i0HDXfCXPLA) **Duration:** 00:07:59 ## Summary - Successful enterprise AI projects are likened to a symphony, where technology tools act as instruments that must be coordinated and guided by a clear “sheet music” (strategy and processes). - Choosing the right infrastructure (on‑prem, cloud, or hybrid) and optimizing it for storage versus compute depends on the specific data types and use‑case requirements. - Data originates from many sources—point‑of‑sale systems, CRM, finance, etc.—and must be integrated across legacy IT environments and newer cloud‑native platforms to support modern analytics. - While rapidly evolving AI/ML tools and large language models generate valuable insights, domain‑savvy business stakeholders are essential to frame the right questions and turn those insights into actionable outcomes. ## Sections - [00:00:00](https://www.youtube.com/watch?v=i0HDXfCXPLA&t=0s) **Untitled Section** - - [00:03:12](https://www.youtube.com/watch?v=i0HDXfCXPLA&t=192s) **Collaborative Data Insight Process** - It emphasizes that business users, IT, and data scientists must work together to harness rapidly evolving technology tools and diverse data sources—collecting, integrating, and analyzing the data—to answer key business questions and create actionable insights. - [00:06:19](https://www.youtube.com/watch?v=i0HDXfCXPLA&t=379s) **Integrating Lakehouse and Data Fabric** - The speaker explains how combining a data lakehouse with a data fabric architecture, alongside proper processes and user involvement, is essential for organizing and extracting value from enterprise data across diverse environments. ## Full Transcript
What are the components that make up a successful symphony?
Well, we first start with
the instruments and who plays these instruments?
Well, it's.
Musicians or people.
People play the instruments.
Now, do they randomly play them?
No.
They follow a process or a type of sheet music that gives them the instructions on how to play the instruments.
Now, it's very similar when we're talking about data and AI
use cases in the enterprise.
However, in this case, our instruments are technology.
It's different tools and platforms.
Now, when we're evaluating different different technology tools, there are different things we need to consider.
There are.
Infrastructure decisions.
Right.
So this could be.
Am I running
On prem,
Am I running
In the cloud,
Or am I running
with a hybrid architecture, right.
This is going to be defined by the requirements that we have for our specific use cases.
Now, the other thing we can also evaluate is how we want to optimize this infrastructure.
Is it optimized for storing data?
Is it optimized for compute or calculating data?
Right.
This is all going to be dependent on the type of data that we're storing and what we're doing with that data.
Now, that data is also created
in different places.
Data can be created, for example, from point of sale systems where customers are buying things.
It could be created from
customer records like a CRM system. It could be created from our finance team.
Right.
That is inputting in metrics about financial performance.
So data is created in a lot of different, in a lot of different places.
Now, the way that all this data is organized can be through either traditionally a legacy IT environment.
So this is what our organization has been using for a long time and has a lot of valuable data locked into it.
However, now there may be new use cases that require
cloud native solutions.
Right.
So depending on the use case, we might need to leverage data from our legacy IT environment, but we might need to run that in a cloud native environment, right?
So both of these have to work together.
Now, once we've done that, there are different tools that we can use to create insights.
So maybe we're using data science and machine learning tools to create algorithms.
Maybe we're using AI and large language models for more creative or complex use cases.
Now, as we all know, the pace of innovation of these different technology tools is rapidly accelerating.
It's almost every week that we see a new model that is more performant, more efficient than the last one.
So, it's no secret that these tools and different technology stacks help us
create value.
Right.
But we need people or stakeholders
to actually make sense of that data.
So now we could have different stakeholders.
We might have
line of business users that,
so these are the folks that understand the domain knowledge of the business and they know what questions to ask.
Right.
We might be asking questions about how do I increase customer satisfaction?
Maybe it's how do I decrease costs, right?
Or maybe I want to try to increase revenue.
Right.
Now, these folks also have to work with.
IT and data scientists.
To create a plan of how to answer these questions
from the data that we have, right.
Now these folks have to work together on these different use cases, which means we require a culture
of collaboration.
Right.
So, what process do these users follow to create these insights?
Well, that's defined in our technology process, right. Now when we're thinking about how data flows through from creation to where it's consumed, well, we first have to
collect the data.
Right. And it might be coming from multiple different sources, such as our edge systems, such as systems in the cloud.
We have to collect all that data, right?
And then we have to go through data integration.
So this is the process of extracting data from where it's created,
cleaning it, transforming it, and making it available for consumption.
Right. So different methodologies that are available here to we have ETL or extract transform load.
There's also ELT, right?
For different types of data.
Now we can choose where we want to land that data.
So, maybe we've, we've chosen to adopt an enterprise data warehouse for
reporting
or analytics use cases.
Maybe we're using
a data lake
for machine learning use cases.
Right.
Or maybe we are leveraging a
data
lake house
to combine both of these into one environment.
Right.
The way that we organize data is critical for evaluating that process.
Right.
So data is being created with our with our technology tools and the and these are tools as well.
Right. But the way that we organize them into these helps our users extract that value from the data.
Now, what about data that isn't in one of these environments, right?
Are we adopting a data fabric architecture to connect to different data sources that are on the edge that are in different environments?
Maybe they're locked away in legacy, in legacy IT environments.
Right.
A data fabric architecture can help us define how we organize the data.
Okay.
So, these are all critical decisions that our enterprise and users must make to get the most value out of the data.
So, we said technology stacks create data, but business users and different stakeholders capture that data, capture that value using this process.
Right.
So, technology tools and processes create value.
But our users and the business captures that value, right?
So, you need all three technology process and people to create the symphony that you want to create with your data in your enterprise.