Key Layers of the AI Stack
Key Points
- Building successful AI applications requires thinking about the entire AI stack—model, infrastructure, data, orchestration, and application layers—rather than just picking a powerful model.
- The infrastructure layer matters because large language models often need GPU‑accelerated hardware, which can be provisioned on‑premises, via cloud services, or through hybrid solutions, and the choice impacts cost and scalability.
- A dedicated data layer is essential to supplement model knowledge (e.g., recent scientific papers) since pre‑trained models have fixed knowledge cut‑offs, and the quality of this supplemental data directly affects solution relevance.
- Orchestration and application layers coordinate complex workflows (splitting queries, fetching data, summarizing, reviewing) and define user interfaces and integrations, influencing the system’s speed, safety, and overall user experience.
Sections
- Building AI Stack for Real‑World Apps - The speaker outlines the essential layers—model, infrastructure, data, and orchestration—required to turn large language models into practical, problem‑solving applications, illustrated by an AI tool for drug‑discovery research.
- AI Stack: Deployment, Models, Data - The speaker outlines the three layers of the AI stack—deployment options (cloud vs. local), model selection (open vs. proprietary, size, specialization), and data sources with processing pipelines—to help AI builders match infrastructure to their needs.
- Application Layer: Interfaces and Integrations - The speaker explains that the AI application layer adds usability through varied interfaces (text, image, audio, data) with features like revisions and citations, and through integrations that allow AI inputs and outputs to seamlessly interact with users' existing tools.
Full Transcript
# Key Layers of the AI Stack **Source:** [https://www.youtube.com/watch?v=RRKwmeyIc24](https://www.youtube.com/watch?v=RRKwmeyIc24) **Duration:** 00:08:53 ## Summary - Building successful AI applications requires thinking about the entire AI stack—model, infrastructure, data, orchestration, and application layers—rather than just picking a powerful model. - The infrastructure layer matters because large language models often need GPU‑accelerated hardware, which can be provisioned on‑premises, via cloud services, or through hybrid solutions, and the choice impacts cost and scalability. - A dedicated data layer is essential to supplement model knowledge (e.g., recent scientific papers) since pre‑trained models have fixed knowledge cut‑offs, and the quality of this supplemental data directly affects solution relevance. - Orchestration and application layers coordinate complex workflows (splitting queries, fetching data, summarizing, reviewing) and define user interfaces and integrations, influencing the system’s speed, safety, and overall user experience. ## Sections - [00:00:00](https://www.youtube.com/watch?v=RRKwmeyIc24&t=0s) **Building AI Stack for Real‑World Apps** - The speaker outlines the essential layers—model, infrastructure, data, and orchestration—required to turn large language models into practical, problem‑solving applications, illustrated by an AI tool for drug‑discovery research. - [00:03:12](https://www.youtube.com/watch?v=RRKwmeyIc24&t=192s) **AI Stack: Deployment, Models, Data** - The speaker outlines the three layers of the AI stack—deployment options (cloud vs. local), model selection (open vs. proprietary, size, specialization), and data sources with processing pipelines—to help AI builders match infrastructure to their needs. - [00:06:56](https://www.youtube.com/watch?v=RRKwmeyIc24&t=416s) **Application Layer: Interfaces and Integrations** - The speaker explains that the AI application layer adds usability through varied interfaces (text, image, audio, data) with features like revisions and citations, and through integrations that allow AI inputs and outputs to seamlessly interact with users' existing tools. ## Full Transcript
Whether you're building an experimental prototype for your own personal use, or creating an
application to power an entire organization, there are key components of the AI technology stack
that you must get right to build AI systems that can do more than just generate answers but solve
real, meaningful problems. Say, for instance, I'm building an AI-powered application to help drug
discovery researchers understand and analyze the latest scientific papers in their domain. Maybe it
starts with a model that I recently heard about that is supposed to be better
at highly complex tasks like that of a PhD researcher. Model is an important layer of the
stack, but it's just one piece of the puzzle. There's also the infrastructure that that model
will run on, because not all LLMs, large language models, can run on
standard enterprise CPU-based servers, and not all are small enough to run on a laptop. So it
matters what infrastructure you have access to and how you choose to deploy it. Next is data
because in this example, the whole point is to help scientists understand the latest papers in
their field. And models typically have a knowledge cutoff date. So if we want to talk about papers
from, say, the past three months, that means we have to provide the AI system with extra data.
That will be the data layer. Next would be the orchestration layer. Because to do
a complex task like this probably is going to require more than simply providing a large prompt
into the AI system and getting an output, a single output, out. Instead, we'll want to break that user
query up into different parts. Help um, plan how the AI solution is going to actually
tackle this problem, what data it needs, and then do the summarization and creating an answer and
maybe even review that answer. Finally is the application layer. And this is because at the
end of the day, there's a user using this tool. So there will have to be an interface that defines
what the inputs will be and what the outputs will be. It might not be as simple as text in and text
out. And there's also the issue of integrations. So, will the actual results of this be something
that's integrated into other tools that this user uses? It's important to understand
all the layers of the AI stack, whether you're building a solution from scratch or using
solutions which might manage several of these layers for you as a service. This is because
across the stack, from the hardware, all the way up to the user interface level, the choices you make
will have important implications on your solution's quality, itsspeed, its cost and its
safety. When it comes to infrastructure, LLMs generally require AI-specific hardware,
specifically GPUs, and these can be deployed in one of three ways. The first would be on premise,
that is, assuming you have the means and resource to buy this kind of infrastructure yourself.
Second option would be cloud, and that would allow you to rent this capacity and be able to scale it
up or down as needed. Finally would be local, which usually means on your
laptop. Not all lap ... laptops can support LLMs of different sizes, but there are certainly LLMs on
the smaller end of the range that can be run on the kind of GPUs available in a standard laptop.
The next layer is models. So AI builders have plenty of choice when it comes to what model they
can use. One dimension to consider is whether the model is open
versus proprietary. Another dimension is the model
size. So we have large language models; we also have small language models that might
be lighter weight and able to fit on more lightweight hardware,uh, but might not have exactly
the same thinking capacity as a large language model and instead be specialized for more
specific things. Finally is specialization.
Which sometimes goes hand in hand with size. Some models might perform better on things like
reasoning or tool calling or generating code. Others might have different language strengths
than others. There are plenty of new models over 2 million already in model
catalogs, like Hugging Face that can serve any mix of these different needs that an AI builder might
have. The next layer of the stack is data. This breaks up into a few different components,
so the first would be data sources themselves to supplement the model's knowledge.
This could also include the pipelines to do any processing,
pre-processing, post-processing of that data, as well as vector
databases you may use. Or retrieval systems,
also known as RAG. Vector databases is the step where that external data is actually vectorized
into embeddings that are saved so your model can retrieve that context more quickly and augment it
with this additional knowledge that the base model does not have. That's important because base
models are usually trained on publicly available information, which might not always be complete to
accomplish the task that you have. You might need to supplement with additional data. The next layer
is orchestration, because building an AI system that does something more complex than
just generating text or answering questions requires breaking the initial user input down
into smaller tasks. Those can start with things like thinking,
using the model's reasoning ability to plan out how it will tackle the problem. That
can also include things like execution, where the model does tool calling
or function calling, as well as steps like
reviewing, where an LLM can actually provide its own critique of the initial generated
responses and initiate feedback loops to even improve those responses. This layer
is very quickly evolving, with new protocols like MCP and new architectures for how to best
orchestrate increasingly complex tasks.
Next is the application layer, so the most widely used AI systems do follow a pretty simple design of text in and text
out. But as we use these tools in our work in life, there are important features that become critical
for the actual usability of AI and these factors make up the application layer.
First factor is interfaces.
The most classic interface is text in and text out,
but there are other modalities that can be very valuable for certain tasks too, like image, audio,
numerical data sets, and plenty of other custom data formats.
Also, in the interface,
it's really important to keep in mind the ability to do things like revisions or
citations so that when the user sees what the model comes up with, they have the ability to edit
that or inquire on it further. The second consideration is integrations, and that
comes both in the form of integrations, of allowing other tools that the user uses to
actually send inputs to the AI system, or to take the
model outputs and automate how that gets integrated into some of the tools that they use
in their day-to-day work. All together, these layers of the AI stack, from the
hardware to the models, the data you use, how you orchestrate it, and the application and the
usability of it, matter because when we have a clear understanding of how they fit together, we
can see what's truly possible and make practical choices to design AI systems that are reliable,
effective and aligned to our real-world needs.