Red Team Tackles AI Threats
Key Points
- AI introduces an entirely new attack surface, requiring security teams to continuously learn and adapt to novel threats rather than treating it as a one‑time testing effort.
- Chris Thompson leads IBM X‑Force’s Red Team, which comprises about 180 hackers who focus on advanced penetration testing for high‑value targets such as banks, defense contractors, and nuclear facilities, and they actively share tools and research with the wider security community.
- Rapid AI adoption across products is outpacing proper risk assessments, leading many organizations to deploy generative AI and machine‑learning systems with insecure configurations, such as missing authentication and unsafe code execution in production.
- These misconfigurations allow attackers to access sensitive assets like enterprise data lakes, execute arbitrary code, and bypass traditional security controls, amplifying the potential impact of AI‑related breaches.
- Ongoing vigilance, thorough security testing, and a mindset that treats AI systems as continuously evolving targets are essential to mitigate the heightened risks introduced by generative AI.
Full Transcript
# Red Team Tackles AI Threats **Source:** [https://www.youtube.com/watch?v=HZYBj-zeUlY](https://www.youtube.com/watch?v=HZYBj-zeUlY) **Duration:** 00:22:27 ## Summary - AI introduces an entirely new attack surface, requiring security teams to continuously learn and adapt to novel threats rather than treating it as a one‑time testing effort. - Chris Thompson leads IBM X‑Force’s Red Team, which comprises about 180 hackers who focus on advanced penetration testing for high‑value targets such as banks, defense contractors, and nuclear facilities, and they actively share tools and research with the wider security community. - Rapid AI adoption across products is outpacing proper risk assessments, leading many organizations to deploy generative AI and machine‑learning systems with insecure configurations, such as missing authentication and unsafe code execution in production. - These misconfigurations allow attackers to access sensitive assets like enterprise data lakes, execute arbitrary code, and bypass traditional security controls, amplifying the potential impact of AI‑related breaches. - Ongoing vigilance, thorough security testing, and a mindset that treats AI systems as continuously evolving targets are essential to mitigate the heightened risks introduced by generative AI. ## Sections - [00:00:00](https://www.youtube.com/watch?v=HZYBj-zeUlY&t=0s) **Inside the AI Security Mindset** - The conversation examines how AI creates a novel attack surface that demands continuous learning and testing, with insights from IBM X‑Force’s red‑team leader and an AWS senior security architect. ## Full Transcript
being in security you're constantly
having to learn new technology so that
you can break it so you can understand
how it works under the Ood and AI is is
no exception the the biggest piece for
us is it it's not just you know a new
web application framework or you know a
new web server technology it's a brand
new attack surface with a lot of really
uh interesting new new attacks that can
be conducted against it and so it's not
something that you test months and
you're
done no nobody's interested in security
until something breaks and then it's all
we can think about so on today's episode
I want to get inside the AI security
mindset with two people who know it best
one is a real life hacker who just
happens to lead a security team at IBM
called xforce yeah you heard what I said
xforce the other a senior security
architect at AWS joining me now Chris
Thompson what's up Chris hey thanks for
having me Chris welcome to the show I
got to ask you how did you get here I've
had the great privilege and honor to
build the uh adversary services or the
red team at xforce Red over the last
eight years um so that team is really
focused on hacking into big Banks or
defense contractors or nuclear power
plants and and you know showing how we
can achieve objectives and not get
detect it can you tell me a little bit
about about the people that you work
with there at X Force raid we have over
I think 180 hackers here everyone
contributes whether that's in
vulnerability management or application
security testing we're on the red team
last year for example we spoke at
blackout security conference six times
um we put out some really Innovative
post exploitation tooling um a lot of
open-source security testing toolkits
for the community to use Chris we've
we've actually seen some things get
broken recently you know a lot of big
security breaches how has Genai changed
hackers are doing and how you work being
in security you're constantly having to
learn new technology so that you can
break it so you can understand how it
works under the hood and AI is is no
exception the the biggest piece for us
is it it's not just you know a new web
application framework or you know a new
web server technology it's a brand new
attack surface with a lot of really uh
interesting new new attacks that can be
conducted against it and so it's not
something that you test and you're done
especially the risk is compounded
because of how many companies are
rushing to slap AI into every product
Under the Sun and they're trying to you
know have that first mve mover advantage
and they're not pausing to really
perform proper risk assessments so we're
finding when we test these these gen
applications and these traditional ml
you know use cases is we're we're seeing
a lot of these environments have been
set up without proper
authentication um they're allowing codec
execution in the backend um production
environment they're allowing us to get
access to Enterprise data lakes and you
know it's really just a symptom of
people trying to move way too fast um
and worry about security problems later
what does a day look like at X Force
typically you know we we would hear from
a customer that it's about to roll out
you know a new financial application or
new foreign exchange platform or maybe a
new banking platform and they'd say hey
we're we're integrating this chatbot so
a customer can prer a wire transfer or
look up account history for example and
so we're looking at from a traditional
web application perspective what are
these traditional attack paths that we
need to be concerned of and what are the
new uh types of attacks that we need to
worry about when these multimodal chat
Bots and these multi-turn chat Bots that
that are interfacing with different apis
in the back end so we we need to figure
out hey how how is this functionality
being called couldn't I talk
really ask this chatbot to load um
insecure code from the internet could
they build you know some sort of code
execution flow with within the B chatbot
itself um and so there's a lot of of new
attacks that we really need to be aware
of and threat model as we're testing
these applications to ensure that you
know we're not seeing customers end up
in the news when these companies are
rushing to integrate AI you mentioned
just how dangerous it can be if you move
too quickly with it what percentage
would you think are actually secure
that's really tough of I I think there's
there's so many components to leveraging
AI into a solution you have the models
that go into these applications that are
being trained and tuned and you know
downloaded from hugging face and
evaluate it you have the model training
and tuning environment itself you have
the whole m SE Ops pipeline which is new
what's your first step when you're
starting to hack a new gen project it's
taking a kind of a risk-based approach
to threat modeling so is this just a a
you know a sales chap bot that's going
to help somebody you know pick a widget
off of a of a a storefront or is this
you know a healthc care application
that's going to allow you to query
previous um diagnosis from your medical
history right and so we start to think
about how how sensitive is the data
that's being pulled into this
application from Enterprise production
environments and then what um you know
different systems is this interfacing
with is this making calls to like an
ethic Healthcare database is this making
calls to like a customer relational
management system how are those apis
secured you know what's being exposed
publicly what's privately what can we
hit from within the Gen solution and
then we come up with a testing plan we
say okay you know this this type of
testing is appropriate for this whereas
you know this this we can spend a little
less time on we also try to push a lot
of teams to not just think about the
front-end solution because we're seeing
a lot of of code execution and platform
compromise and and you know a lot of
it's private and not hit the news and we
just happen to hear about it from
different customers or from Partners we
think it's really important that that
customers start to think about you know
again that endtoend flow of how did we
evaluate these models how do we know
about the AI supply chain and kind of an
sbom or software build materials that is
part of the solution where are those
python packages being downloaded from
you know how is who rolled out my AI as
a service platform and who configured it
and who enabled logging and really
helping customers to challenge
assumptions around all of that kind of
backend development production
environment and flow so that way in the
future when you know another uh AI
supply chain attack happens and some
python package got back doored for
example customers can quickly say okay I
know exactly what I used in my solution
and I don't have to worry about it
obviously AI is still a very new field
so I I need you to be honest with me on
this one from your POV do we actually
have the people or the skills that we
currently need in order to properly
secure AI like is there a skills Gap
somewhere there's a massive skills Gap
if you went on LinkedIn three years ago
um the amount of people that said they
were data scientists or AI Security
Experts versus three months ago is night
and day right everyone's claiming to to
be an expert in this field or they're
rushing to to train up and and the
reality is there's just a massive skill
shortage there's already huge skill
shortage and security already and now
this is a new you know brand new space
that we need to be constantly you know
upscaling our our both our internal
security staff but also you know making
developers and data scientists and
management aware of those those
different security risks and and making
sure that you know we're we're
partnering with the right people that
come help test or we building out that
skill set in house I'm I'm trying to
figure out whose responsibility It
ultimately is though like is it the
developers responsibility to be thinking
about security as they go about building
their projects or is it somebody else's
yeah great question I think it comes
down to a shared responsibility model
and having those discussions with your
vendors and your different developers
and your data scientists you know a lot
of people that are training and tuning
these models they say that the security
of the model is not their responsibility
and you know I liken that a lot to you
know a web application developer saying
I don't need to code an application
securely just put a web application
firewall in front of it right it's awaf
is not a good replacement for secure
code just as as a you know AI firewall
or guard rails are not you know a good
solution for securing models so it's a
shared responsibility for whoever is
training and tuning those models or
supplying those models it's shared
responsibility with the develop Vel
opers and the production teams to build
out you know decent guard rails so you
know they might not prevent every attack
but you can at least you know have a
canary in the coal mine and know if your
applications attempting to be abused or
whatnot or your Solutions being abused
and then it's you know having those
conversations with your your AI as a
service vendors like you know Azure ml
or Watson X or AWS sagemaker you know
what testing has gone into this platform
how do I show Providence of the models
and and the the AI supply chain and that
you know that esbon how do I know what
testing you've done on this platform or
this this application that you've built
for me and what expectations do you have
of myself for testing my my application
as it goes live since you are a hacker
and I'm assuming that you enjoy your job
do you get excited when you find a gap
absolutely um it's what we live for as
much as we love you know breaking stuff
and finding apps and and circumventing
all those controls it's also you know as
you are in the industry for a while you
kind of get tired of only burning things
down and so starting to have those
relationships with the blue team and
with the developers and saying hey we're
tired of finding this bug in every
single application you create you know
let's take a look earlier on in the
development life cycle let's let's solve
this from the start so it doesn't make
its way into every app these are the
controls you should put in place or
these the detections you should have and
if if those primary controls fail here's
how you should be you know approaching
this well then what's your favorite
thing that you've ever hacked Chris
definitely some breaking into some uh
military bases back in the day was was
was a lot of fun we've haded major major
sporting events um most of the big Banks
and and you know exchanges in the world
uh nuclear power plants defense
contractors chemical
manufacturers it's it's uh Telos uh law
enforcement intercept systems everything
you name it we've we've hacked it and
I've had the pleasure to lead you know
the majority of of uh big engagements
well Chris thank you so much for joining
us today it's funny I feel like I felt
my anxiety Spike some when you were
talking but then I also felt relieved at
the very end when I like oh but that's
okay he's one of us he's one of the good
guys so um we really appreciate you
thank you for that Insight I feel like
we could really just talk about hacking
all day so now knowing what we're
dealing with in terms of risks let's
find out how you deal with it from aws's
Resident security expert Mita Saha
welcome to the show thank you Albert so
let's get straight to it how has gen aai
changed your job it has definitely
revolutionized the whole security
landscape it has introduced both
challenges and opportunities
specifically in my job I know that now I
first need to understand how gen works
so that I can understand how can I
secure it we also need to make sure
given as an architect when I'm trying to
design new Solutions or trying to do a
migration I need to be more conscious
about the tools I I'm using am I able to
leverate geni for an efficiency in my
current work that is one landscape where
we are trying to learn gen learn the
benefits of gen and leverage it so it
has definitely impacted I would say in a
more positive and exciting way because
now I'm trying to learn new things and
implement it as the same time can you
give me an example or two about the kind
of challenges that gen can pose you know
through the hands of Bad actors for you
but then also how you've been able to
mobilize the power of gen gen has
revolutionized to hold hacking landscape
like on one hand it introduced new
attack vectors and Amplified the
existing threats generating more
convincing fishing emails generating
more def fake audios or even crafty
evasive Mals on the other hand AI power
tools can automate and streamline
vulnerability detection vulnerabilities
has always been there in our security
landscape rather in the industry be it
any industry
so we can now leverage the AI power
tools and it can help us automate and
streamline the vulnerability detection
process it can help us enable faster and
more comprehensive security assessments
we cyber Security Professionals must
adapt rapidly employing AI for defense
while simultaneously we have to mitigate
the risks possessed by the AI driven
threats what's the most common mistake
that people make when they're securing
their data or their AI as I have been
fortunate to also write a white paper
have been part of a white paper we have
collaborated uh featured a few months
ago on securing gen and we have stated
in our white paper that only 24% of the
current gen projects have a security
component in it even though
81% of Executives say secure and
trustworthy AI is essential as you can
tell this suggests that many
organizations are not prioritizing
security for their AI initiatives from
the start that is a potential oversight
like nearly um 70% of Executives say
Innovation takes precedence over
security when it comes to gen now
deprioritizing Security in favor of
innovation could lead to some
vulnerabilities the number one mistake
that we can surely fall into is
neglecting the security fundamentals or
having an immature security culture in
our organization which can leave our
organizations ill prepared to address
the conventional threats like malwares
social engineering that take new forms
with chaii you talked about prioritizing
Innovation over security so with that
prioritization let's say that some of
the companies that are listening to us
right now um either did do that right
they've rushed to get AI ready and they
bypassed the security they thought you
know what we'll just do this later what
can they do now what's the first thing
that you would advise that they do in
order to in order to Rebound in order to
rectify that situation one immediate
step that we can take is to conduct a
comprehensive security audit risk
assessment best in my opinion will be to
pause the deployment right now and then
conduct the security assessment instead
of doing them in parallel but if in some
organization in some business if that is
not a possibility then at least just run
the security assessment in parallel to
whatever you're doing right now this
audit should be performed by a team of
cyber Security Experts can be from your
own organization or you can hire from
any of the other cyber security forms uh
that you trust in and who can identify
the vulnerabilities the potential attack
vectors the security gaps and the a area
of
non-compliance with your industry
standards and the AI initiative that
your organization have taken second will
be conducting the right threat modeling
to uncover the security gaps in your AI
environment and to determine how the
policies and controls need to be updated
to address those New Gen threats then
conducting right kind of penetration
testing to simulate those attacks and to
identify the potten itial
vulnerabilities that can be exploited
and then finally I would say evaluating
the systems to understand how the data
is handled and the data handling
practices like Access Control encryption
mechanism and other potential points
like networking and the most important
part of this assessment should be a
detailed report as a security um
engineer consultant myself I believe
that if I just come to my customer or my
partner with the problems that won't
help them right so I think a a key
output of this assessment should be a
detailed report I should be able to
guide or recommend I'm not saying you to
do this and that but I can give you a
recommendation if I'm doing an
assessment in your experience what's the
number one potential AI related attack
that's most likely right now and how can
that be prevented if I have to call out
one top AI related attack that concerns
our SE Suite which which I have read on
is uh adversarial attacks on the AI
systems these attacks involve
manipulating the input data to an AI
model to deceive the model in a way that
it causes the model to make incorrect
predictions or
classifications and generate harmful
outputs these attacks exploit the trust
and Reliance we have played based on the
AI systems and it can have severe
implications on the security and
integrity of the systems that
organizations depend on so this is going
to be a rapid fire round where I'm going
to list some different scenarios objects
I guess and I'm going to ask you just
for a simple yes or no for each of these
as I call them on out using geni hackers
can compromise the following your phone
phone yes your TV oh yes your it system
yes oh your car yes depends how how
advanced I have uh built my car but yes
oh well me this is a lot okay your home
internet kind of um I am trying to say
no but again if I don't set up the right
measures then yes okay you're
refrigerator not right now but again yes
if I is using a iot the iot version of
my refrigerator and finally your dog
oh you know what yes probably
yes and I can give you a use kiss for
all of them but yeah how can J AI hack
your dog so de fake audios say I'm
working and I am not always home with my
dog and I have a device through which I
can talk to my dog when I'm not at home
now if that device is compromised and
compromised with de fake audios the J
bot can actually fake my exact voice
just not my dog even my friend my mother
who knows me very well right and if my
mother gets a call not from my phone
number because my phone is with myself
so the J I bought will not have my phone
number but say if I'm calling from a
different device and that is the Deep
fake audio they do and if my coming back
to the dog's reference if I am saying
something to my dog and my dog is very
well behaved listens to me and I tell
him or her to do something in the house
which can become very scary I can just
say just go to the backyard and go to uh
a neighbor's house and get out of the
house so deep fake audios is definitely
something to look out for so that's why
we should be always
evaluating whoever is calling us or how
it is happening so okay well we're
relieving this feeling a lot more
confident I
guess that was I think that you've given
me so much to to chew upon right here so
I thank you so much and in fact Mamita I
appreciate all the time that you shared
with us today this has been beyond
insightful I'm going to speak on behalf
of the listeners and viewers if I may
and say like whoa you know you you've
given us a lot to look out for but then
also you've given us some great
confidence and faith in what can be done
utilizing geni appreciate you for that
now again I also want to give a shout
out to Chris thank you Chris for joining
on today's podcast and friends that's it
for today so thank you all for listening
thank you for watching and of course if
you have thoughts please post them in
the comments below and I promise we'll
see you again soon
[Music]