Choosing Enterprise LLMs: IBM Granite
Key Points
- Enterprise‑grade foundation models should be evaluated on three core metrics: performance (latency/throughput), cost‑effectiveness (low inference energy and expense), and trustworthiness (low hallucination and clear training‑data provenance).
- Trust is especially critical because generative AI workloads can consume 4–5× the energy of traditional web searches, so models must balance high performance with minimal inference cost while offering transparent, auditable training data.
- IBM’s Granite foundation models are positioned to meet all three criteria equally, delivering strong performance, competitive operating costs, and built‑in trust mechanisms such as documented data sources and reduced hallucination risk.
- The Granite models are open‑source (Apache 2.0) and trained on vetted enterprise‑grade datasets—including 1.8 M scientific papers, all U.S. utility patents (1975‑2023), and public‑domain legal opinions—totaling about 6.5 TB of data, providing unprecedented transparency and relevance for business applications.
Full Transcript
# Choosing Enterprise LLMs: IBM Granite **Source:** [https://www.youtube.com/watch?v=cVDv9apGTXo](https://www.youtube.com/watch?v=cVDv9apGTXo) **Duration:** 00:06:32 ## Summary - Enterprise‑grade foundation models should be evaluated on three core metrics: performance (latency/throughput), cost‑effectiveness (low inference energy and expense), and trustworthiness (low hallucination and clear training‑data provenance). - Trust is especially critical because generative AI workloads can consume 4–5× the energy of traditional web searches, so models must balance high performance with minimal inference cost while offering transparent, auditable training data. - IBM’s Granite foundation models are positioned to meet all three criteria equally, delivering strong performance, competitive operating costs, and built‑in trust mechanisms such as documented data sources and reduced hallucination risk. - The Granite models are open‑source (Apache 2.0) and trained on vetted enterprise‑grade datasets—including 1.8 M scientific papers, all U.S. utility patents (1975‑2023), and public‑domain legal opinions—totaling about 6.5 TB of data, providing unprecedented transparency and relevance for business applications. ## Sections - [00:00:00](https://www.youtube.com/watch?v=cVDv9apGTXo&t=0s) **Evaluating Enterprise Foundation Models** - The speaker outlines three key criteria—performance, cost‑effectiveness, and trustworthiness—for selecting an enterprise‑grade large language model, citing IBM Granite as an example. ## Full Transcript
when it comes to picking a large
language model with sport for Choice
last time I checked there was something
like
700,000 different llms or large language
models on hugging face now I'd like to
cover just a couple of those
specifically the IBM Granite Foundation
models but first let's consider how to
pick an Enterprise grade Foundation
model meaning an nlm suitable for
deployment in an Enterprise setting
something you'd be happy to run your
business with so let's consider that
through three different metrics so the
foundation model it needs to be
performant that's an important metric
but it also needs to be cost effective
and it needs to be trusted those are the
three metrics we're going to consider
and trust it of course because you can't
scale generative AI with models that you
can't trust so take these one by one now
by performance we're talking about
measurements like latency and throughput
is a foundation model able to keep up
with the speed and Enterprise requires
it to operate that then related to that
is cost Effectiveness now according to
the scientific Jour nature a search
that's driven by generative AI will use
something like 4 25 times the amount of
energy that's needed to run a
conventional web search so we need a
foundation model that can deliver the
necessary performance with low
inferencing costs and we need the
foundation model to be trusted and we
can gauge that through metrics like
hallucination scores but also a model
that offers transparency so we know what
data the model was trained on and I
think in many instances models are kind
of SK
a bit like this they're highly
performant but they're expensive to run
at inference time and there's a lack of
transparency on the training data the
model was built with now with the
granite models IBM set out to create
Enterprise grade Foundation models that
apply an equal weight to all three of
these metrics so it looks more like this
so what should you know about the ABM
Granite Foundation models well many of
the models are open source you can find
them on hugging face under the Apache
2.0 license that enables broad
commercial usage now these models also
have transparency in training data
meaning we actually know the data
sources that we use to train the models
and that's quite atypical most llms are
uh notoriously vague on how their models
were trained so that's a nice change now
Granite language models are trained on
trusted Enterprise data spanning
academic code legal and finance data
sources as such as well the first 13
billion parameter Granite llm was
trained on about
6.5 terabytes of data and that includes
1.8 million scientific papers that were
posted on archive it ALS o includes all
us utility patents granted by the
USPTO and that's from 1975 all the way
through to 2023 and it includes the
public domain free
law which are legal opinions from US
federal and state courts essentially the
models have been governed and filtered
to only use Enterprise safe data sources
the granite models have also been
designed to to be performant as well
especially in areas of coding and
language tasks outperforming some models
that are actually twice their size and
smaller models means also they're more
efficient with less compute requirement
and a lower cost of inferencing now I
keep mentioning the granite models
plural so which models are we talking
about so Granite is actually a family of
llm Foundation models spanning multiple
modalities and you can find many of
these on hugging face so let's take a
look at some of them and we'll start
with granite for language now these are
decoder models of different parameter
sizes so that includes a
7B open source model and the B here
refers to billions of parameters so
seven billion parameters there's also an
8B model that's designed specifically
for Japanese text there's a couple of
13B models and there is a 20 billion
parameter multilingual model that
supports English German Spanish French
and
Portuguese now there's also Granite for
code and that again comes in different
parameter sizes from 3 billion all the
way through to
34 billion parameters and granite for
code is trained
on6 programming languages now there's
also Granite for time series that's a
family of pre-trained models for time
series forecasting these models are
trained on a collection of data sets
spanning a range of business and
Industrial application domains and these
models are optimized to run on pretty
much anything even a laptop and then
finally there is granite for geo spatial
which is a partnership between NASA and
IBM to create a foundation model for
Earth observations using large scale
satellite and remote sensing data so
that's the IBM Granite models models
that are trusted performant and
efficient and that can be applied to a
wide variety of Enterprise use
cases for