Trust, Transparency, and Governance in AI
Key Points
- Trust is identified as the foremost prerequisite for deploying large‑scale generative AI in enterprises, as without confidence in model outputs the technology’s benefits cannot be realized.
- The speakers highlight the prevalence of AI “hallucinations” and other toxic behaviors (e.g., bullying, gaslighting, copyright violations, privacy leaks) that erode trust and create fear among organizations.
- Kush Varshney’s extensive background—hundreds of publications, open‑source fairness and explainability toolkits, a book on trustworthy machine learning, and leadership roles at IBM and the MIT‑IBM Watson AI Lab—underscores the depth of research effort behind trustworthy AI initiatives.
- The discussion points out a gap between the rapid push to operationalize generative AI and the need for robust governance frameworks that address reliability, transparency, and ethical risks.
- Real‑world examples, such as chatbots producing a mix of accurate and fabricated information, illustrate the concrete challenges of ensuring AI honesty and the necessity of systematic mitigation strategies.
Sections
- Introducing Trustworthy AI Leadership - In the opening of the AI Academy series, Kate Soule greets distinguished researcher Kush Varshney to highlight the paramount importance of trust, transparency, and governance for enterprise adoption of generative AI, emphasizing his extensive work and public impact in the field.
- Navigating Trust in Generative AI - The speaker stresses that AI development should prioritize trust, transparency, and governance, highlighting new risks like hallucinations, data leakage, and bullying that emerge with massive generative AI datasets.
- Open Kitchen Analogy for AI Transparency - The speaker argues that AI systems need transparent, open‑concept “kitchens” revealing data sources, processing steps, testing, and auditing to build trust, and connects this need to fairness concerns such as stereotyping and toxicity that disproportionately harm vulnerable populations.
Full Transcript
# Trust, Transparency, and Governance in AI **Source:** [https://www.youtube.com/watch?v=odrD0OLPeiY](https://www.youtube.com/watch?v=odrD0OLPeiY) **Duration:** 00:09:21 ## Summary - Trust is identified as the foremost prerequisite for deploying large‑scale generative AI in enterprises, as without confidence in model outputs the technology’s benefits cannot be realized. - The speakers highlight the prevalence of AI “hallucinations” and other toxic behaviors (e.g., bullying, gaslighting, copyright violations, privacy leaks) that erode trust and create fear among organizations. - Kush Varshney’s extensive background—hundreds of publications, open‑source fairness and explainability toolkits, a book on trustworthy machine learning, and leadership roles at IBM and the MIT‑IBM Watson AI Lab—underscores the depth of research effort behind trustworthy AI initiatives. - The discussion points out a gap between the rapid push to operationalize generative AI and the need for robust governance frameworks that address reliability, transparency, and ethical risks. - Real‑world examples, such as chatbots producing a mix of accurate and fabricated information, illustrate the concrete challenges of ensuring AI honesty and the necessity of systematic mitigation strategies. ## Sections - [00:00:00](https://www.youtube.com/watch?v=odrD0OLPeiY&t=0s) **Introducing Trustworthy AI Leadership** - In the opening of the AI Academy series, Kate Soule greets distinguished researcher Kush Varshney to highlight the paramount importance of trust, transparency, and governance for enterprise adoption of generative AI, emphasizing his extensive work and public impact in the field. - [00:03:13](https://www.youtube.com/watch?v=odrD0OLPeiY&t=193s) **Navigating Trust in Generative AI** - The speaker stresses that AI development should prioritize trust, transparency, and governance, highlighting new risks like hallucinations, data leakage, and bullying that emerge with massive generative AI datasets. - [00:06:16](https://www.youtube.com/watch?v=odrD0OLPeiY&t=376s) **Open Kitchen Analogy for AI Transparency** - The speaker argues that AI systems need transparent, open‑concept “kitchens” revealing data sources, processing steps, testing, and auditing to build trust, and connects this need to fairness concerns such as stereotyping and toxicity that disproportionately harm vulnerable populations. ## Full Transcript
[AI Academy]
[Trust, transparency and governance in the age of generative AI]
[Chapter 1 - Introduction to trustworthy AI]
Hello, and welcome to AI Academy.
My name is Kate Soule.
I'm a Business Strategy Senior Manager at IBM Research
and the MIT-IBM Watson AI Lab,
and this is my colleague,
Distinguished Research Scientist Kush Varshney.
Kush is an AI researcher with a focus on trustworthy AI.
Kush, I'm really excited we get
to have this conversation today.
I've been working with clients
and thinking about trustworthy AI
from a business perspective for a while now,
but I know you've been innovating in trustworthy AI
from a research perspective for a number of years.
When it comes to AI, I think you and I can both agree,
trust is the number one important thing.
Yeah, it has to be.
If we don't have that trust in those models
that have billions of parameters and they're really huge,
but until we have that trust,
we can't really get the benefit of that AI in enterprises.
Now, you have quite a few accomplishments
to your name in this space, right?
You've published hundreds of papers, you have algorithms
that are working in labs around the world.
You're a sought-after speaker, right?
Yeah.
And I say this to emphasize
that you have a big footprint in this space,
a public footprint in this space,
and given your public accomplishments,
I thought it might be interesting
if I ask some consumer chatbots to learn a little bit more
about some of the work that you're doing in trustworthy AI.
Yeah, that sounds like a fun thing to do.
So you published a book on trustworthy machine learning?
Yep, that's absolutely correct.
You were named an Elevate Fellow
by the government of Ontario, Canada.
Um, I've never heard of that fellowship.
You're a co-founder
of the Machine Learning for Good Social Foundation.
That's almost right.
So I did found the IBM Science for Social Good initiatives,
so we're close.
You've created many open-source toolkits.
So we created the 360 toolkits around AI fairness,
360 AI Explainability, 360 and some others, yep.
You have a PhD in electrical and computer engineering
from the University of Illinois at Urbana-Champaign.
I went to MIT.
So Kush, what's going on
with these chatbot responses here?
Some of these are right,
and some of them are complete fiction.
What's going on?
So I would call that hallucination,
and so that means that these AI systems,
they'll make some things up,
they'll make associations that aren't exactly correct,
and I think that's what happened in our last example.
So it kind of created this association
that didn't exactly exist.
Got it.
I think everyone is feeling the pressure
of operationalizing generative AI as fast as possible,
but when companies hear about AI hallucinating
or other toxic behaviors like bullying or gaslighting,
and there's other concerns around generative AI
in copyright infringements
or the revealing of personal or private information,
and it makes companies concerned
and nervous and even fearful
about adopting generative AI in their organization.
Yeah, and what we have to remember
is that AI is not a race.
It's a journey.
We have to be careful, and as anything
that we want to get into enterprise AI,
it has to have these principles of trust
and transparency throughout.
We have to slow down,
put in all of these governance aspects,
make sure that we're putting in safeguards, guardrails,
and just doing the right thing.
[ Chapter 2 - New risks with generative AI]
I know you and your team
have worked on this for a while, right?
How have the risks changed with the advent of generative AI
compared to the risks we were seeing before
with traditional machine learning?
Yeah, so predictive machine learning and generative AI,
they're kind of two sides of the same coin,
so a lot of the techniques are very similar,
but there are differences.
So the hallucination that you mentioned,
the leakage of private information, the bullying--
all of those are new risks that we haven't seen before.
We still have a lot of other risks as well
that kind of carry over,
but the difference mainly is around the solutions.
How do we address these issues?
And a lot of the reason we can't apply
the same techniques from before
is because of the huge data that we're dealing with now.
It's just humongous, humongous datasets.
Yeah, can you talk a little bit more
about that, specifically?
So when we have these huge volumes of data,
how does that impact our ability to trust a model?
Yeah, the data is so huge.
We can put in data governance techniques.
We can ensure that certain sites are not scraped,
that certain filtering is done and so forth,
but it's beyond the ability of any individual human
or a team of humans to even read through
every single piece of content,
so that's where the challenge comes from.
[Chapter 3 - Elements of trustworthy AI]
Now, let's take a step back for a second
and talk about trust as a concept.
When I talk to clients about trust,
most of the time, their minds jump straight to accuracy,
thinking about quality, and can they trust the model
in the use case that they're trying to deploy it in.
How do you define trust?
Yeah, so I think the starting point is that,
so the quality, the accuracy,
just the general performance of these models,
because without that, nothing else follows,
but that's just the starting point, right?
Yeah.
So there's all sorts of other considerations,
whether it's reliability and robustness or fairness.
Can we as humans understand how the model is working.
Can we understand the entire process
of how it came together.
Can we ensure that the models, these AI systems,
are working for our benefit, not doing something else.
Yeah, I think a valid criticism of AI in general,
including generative AI,
is that it can be a bit of a black box.
Can you speak a little bit more about transparency
as a dimension of trustworthy AI?
Transparency says it, I mean, already, right?
So we think of these AI systems,
they're black boxes in some capacity,
and what we need is more openness.
We need to shed light on them.
And what transparency allows us to do
is kind of understand what's going on from beginning to end.
So an analogy to that is, let's say you're at a restaurant
and it has an open-concept kitchen.
You can see all the ingredients before they're chopped up.
You can see what the chef is doing,
and all of that gives you confidence
that there's just general goodness happening.
And the same thing applies to AI systems.
If we can know where the data came from,
what sort of processing steps were performed,
what sort of testing was done,
what sort of auditing was done, all of that together
gives us the understanding of what's going on.
[Chapter 4 - Fairness, bias and governance]
Now, Kush, you and your team
have also spent a lot of time thinking about fairness.
Can you speak a little bit more about that?
Yeah, fairness is a topic I'm really passionate about,
and in the traditional machine learning sense,
we talked about fairness for, like, hiring algorithms,
for lending algorithms, these sort of things,
but when we moved to the generative AI world,
things are a little bit different.
So the thing that we're most concerned about
is stereotyping and other toxicity,
because it's the most vulnerable members of society
that suffer the most when these systems
are actually doing things in a harmful way.
And this is one of the areas
where I feel like generative AI and machine learning
have a lot in common.
At the end of the day, if they're trained on biased data,
they're going to create biased outputs, and
generative AI, for better or worse,
is trained on human-created data,
and humans have conscious and unconscious biases,
and the data that they create can reflect that.
Yeah, absolutely.
And it's the algorithms that just amplify
all of those societal and cognitive biases as well.
So with all these risks and considerations around trust,
how can clients adopt generative AI
in a safe, responsible, and ethical way?
Yeah, I think the only word I need to say is governance,
and AI governance really starts at the beginning.
What is the intended use
of these systems that we're creating?
Where's the data coming from?
Where's it sourced? How are we processing it?
Putting in all these different checks and balances,
and doing all of the testing in deployment as well.
Can we continuously monitor how they're performing
and step in if they go beyond those guardrails?
Absolutely.
I think you put it really well.
When the stakes are high, you need to be able to trust,
but have that trust validated and verified,
and not just trust for trust's sake.
Yeah.
Okay, it's time to wrap up.
Thank you so much, Kush.
And for everyone else,
thank you for watching this episode of AI Academy.
Please join us again for future episodes
as we unpack some of the most important topics
in AI for business.