Understanding Asynchronous Message Queuing
Key Points
- Jamil Spain (IBM Cloud Developer Advocate) explains that traditional application design expects immediate, synchronous processing via REST APIs, which often leads to extensive error‑handling code.
- Message queuing is introduced as an architectural pattern that enables asynchronous communication, allowing different parts of an application to operate independently and remain functional even when other components are unavailable.
- A “message” is simply any piece of information—such as data payloads, files, or metadata—that needs to be transferred from one subsystem to another for processing.
- The “queue” component stores these messages in the order they arrive, delivering them sequentially to the consumer at a later time, effectively decoupling the producer and consumer.
- Email is cited as a familiar example of asynchronous messaging, illustrating how queues let systems handle tasks later, improve reliability, and avoid the tight coupling of monolithic, synchronous designs.
Full Transcript
# Understanding Asynchronous Message Queuing **Source:** [https://www.youtube.com/watch?v=xErwDaOc-Gs](https://www.youtube.com/watch?v=xErwDaOc-Gs) **Duration:** 00:15:37 ## Summary - Jamil Spain (IBM Cloud Developer Advocate) explains that traditional application design expects immediate, synchronous processing via REST APIs, which often leads to extensive error‑handling code. - Message queuing is introduced as an architectural pattern that enables asynchronous communication, allowing different parts of an application to operate independently and remain functional even when other components are unavailable. - A “message” is simply any piece of information—such as data payloads, files, or metadata—that needs to be transferred from one subsystem to another for processing. - The “queue” component stores these messages in the order they arrive, delivering them sequentially to the consumer at a later time, effectively decoupling the producer and consumer. - Email is cited as a familiar example of asynchronous messaging, illustrating how queues let systems handle tasks later, improve reliability, and avoid the tight coupling of monolithic, synchronous designs. ## Sections - [00:00:00](https://www.youtube.com/watch?v=xErwDaOc-Gs&t=0s) **Introducing Asynchronous Message Queuing** - Jamil Spain explains how message queuing lets applications communicate asynchronously, breaking monolithic dependencies and removing the expectation of instant responses. ## Full Transcript
hello my name is jamil spain developer
advocate with ibm cloud and my topic for
today is message queuing
now as a technologist when i first
started out developing applications
there was an expectation that in my code
that applications would
process everything immediately and even
when i first started dealing with rest
apis that's really the expectation that
i make a
front end makes a rest call to a
back-end through a rest api call that
you're expecting a response to come back
some type of response code must go back
to let me know
404 was not good 200 it was good but
again there's an expectation that things
happen instantly and i get instant
confirmation that something is happening
and often i wrote a lot of code to
[Music]
accommodate situations where that
communication could not happen or
or did not function uh properly there
but today's topic kind of branches out
through that let's deal with break down
what the term message cueing actually
means here so overall from a 30 000 foot
view
it's a definitely a architecture
technique that allows you to break
applications apart
apps apart
with what we like to call asynchronous
communication
i'll abbreviate that there
so a secret is communication so i want
my particular applications to or
particular subsets to still function and
not worry about the other parts there
versus the other part where all my code
was in one stack
and uh and only in a monolithic
application or when i build an
application where all the functionality
exists that
you know one set of functions or one set
of code of my application i'm just
always handling a transaction off to
another set of code but there is that
expectation that things follow through
in process right then there's no way to
really kind of delay things to happen
later and this is what message queuing
is all about but let's break down the
first part here first let's deal with
the word message
and as it denotes that
it is really something
that
a piece of information that wants to get
processed from one system to another an
application
um we'll just say subsystem from as well
kind of denoted by um with that
particular piece so you have data
that needs to get
transmitted now this could be actual
data payloads actual files
um it could be metadata that that
references that something should happen
on the other end um so many different
types
of
data
formats can be there
all right and then we have the word q
which q in the most simplest form
refers to kind of a a line of things
that are processed sequentially so when
you think of a message queue think of
the messages of coming into a queue as a
way that kind of holds the messages in
the order they are received and delivers
them to the actual party so one way to
think about this i know this kind of
explain message explain queuing explain
the overall but think about a tool that
everyone uses every day
email email is an
excellent example of asynchronous
communication and how that works i can
theoretically or not theoretically
actually in reality in real life i send
a message out but i can wait for a
response later it goes into an inbox
think of that inbox your email inbox as
a queue of messages that are held there
and at some point
the
person who's going to receive it
has the opportunity to look at it
whenever they want if they take a longer
time there's a larger list you know that
has accumulated of messages that you
have to read which is usually the case
when i go through some of my email
email boxes or for ones that i check
constantly like say for instance your
work email you're always constantly
going through the list so you're there
waiting to kind of reply back but
excellent example of asynchronous i can
really fill out an email compose it send
it off i'm not worried about whether you
received it or when you answer it but i
can move on to something else all right
so perfect example of asynchronous and
we'll carry that example through um as
well
to some of the use cases here uh so we
talk through those let's talk about the
type of messages that occur so we've
mentioned qli which is a
line
of messages
so if i were to draw this out
in in this perspective here let's do it
this way let's take this green as the
queue
and then let's say that each one of
these is going to be an actual message
and so at some point you're going to
have the
they're going to send the kind of the
message here
and then at the other end
somebody's going to open it up get my
little envelope there all right little
artwork here so the person that's
putting the message on is going to be
what's known as the producer
and then the person consuming the
message is the consumer
so these are often the two profiles or
roles that you can take and of course
you can be the producer one time and be
the consumer on another example so this
one pattern here that you're looking at
is more like a point to point i am
sending it from one application to the
other all right
it could be a
let's say i have a web form and you're
filling out and as you complete the web
form i take the metadata from that form
json object
i could put that on a queue to be
processed later to be inserted into a
database to
be inserted to another system for
marketing automation
whatever it may be the use case the
point is that i have i need messages to
guarantee delivery to get there and not
be dependent upon you submit the form i
send it back to that other system and
get a response back
i can keep ingesting
particular messages form submittals and
keep loading those in to go from there
so when you have these kind of use cases
these are excellent things that should
read out that hey instead of me just
doing direct http connection let me
think of a message cueing pattern that i
can facilitate to go here so we'll call
this point to point the next one is
something like a publish subcri
publish subscribe also abbreviated as
pub sub that you will hear and this
works from the capability that i have
a producer
let's go back to the green i'll do
another cue and we'll do more messages
that come in
so on the pub sub model
i'm going to have
multiple
parties or applications that are
interested in getting this message
as it comes through
all right so pub publish subscribe so
these consumers are listening for this
particular i'll call it a channel or
interested in this particular thread of
messages that may come across so it
could be that new customer signs up you
may want to send something to
um for a new order e-commerce order
comes in you may want to send something
to accounting for an invoice to go out
you may want to send something to the
shipping
or notifications for stock or you know
any multiple
things that you want to kind of get
through that could be something you can
shoot out to kind of go and the last one
i'll work through is what we like to
call
request reply reply requests it's done
from the standard that you have a
producer
that's going to
send out
a message
all right
let me redraw this again really fast
here i got messages that i'm going to
put in
okay
and so
this could be one way they could have
one channel
all right so they could be one way where
they want to go out
someone could consume it
i could put a request out
someone gets it
does some processing
but the response to that that they want
to do
i'm going to also be listening
on another queue
for the response to that
so that could be now that all of a
sudden
where they're reading this request that
comes through they then could put the
resulting workload or whatever
computation that they're doing whatever
algorithm that must be done
say for instance maybe your
some of those patterns where you want to
put your information in and look at all
the offers that are out there it could
be an actual search to go out and get
all this information compile that back
to
maybe a list of object that information
comes back and then i'm listening the
producer is also listening to another
queue to say ah the response is ready
and then it takes that information and
then presents it so you have
point to point
all right this is a pub
pops up and then more of a
request and reply all right so different
type of messaging patterns that can
occur and again i love to kind of think
through things from a backwards
perspective it's really about as a
technologist and a solution architect
sometimes you want to put your hat on
where you say
what is my intent what do i need to
happen and that really drives what
particular piece of technology can drive
that particular component here all right
so three patterns we identify message
killing let's wrap it up with the
benefits naturally of why you want to
facilitate or go through the effort to
orchestrate something like this and i
like to bring this down to two major
particular pieces here the first
decoupling
all right it looks like an m but it's
really an eye and an n all right trust
me there all right so decoupling here so
what this allows me to do we mentioned
the first part about breaking
applications apart
it allows me now to have
separate items that can occur so i can
in going along what we like to call the
microservice pattern responsibility
pattern i can dedicate one particular
piece of code application to just be
responsible for that one piece of work
all right
and if i was in a monolithically kind of
code base
i would have to progress things all at
one time if i wanted to just work on my
payments module
i would also have to test everything to
make sure how that affects other parts
here well
in this particular piece let's say i
just wanted the web front end to be the
web front end i don't want you to handle
a lot of the processing and doing all
the database connections from there i'll
let other systems do that part but they
can be very well at just connecting to
my database connecting to my services
and my systems of record anything from
that so decoupling allows you to really
separate things up
where apps
can just be
apps you know just
just perform their particular job
here as well and naturally
that will
enforce scalability in your application
because now you have apps that are
decoupled you have teams that can work
on applications i can think about my
application
much simpler i may have one component
that i want to work on at a time while
the others are guaranteed to function
as is and also it allows me to add more
resources to certain parts as we
mentioned in the other use case earlier
where i have a web form that wants to
maybe
handle a lot of high traffic of
submitting or my front end may have a
lot
but i may want to wait before i
process a lot of that on the back end
and not have that immediate
i mean want to put it through more
checks maybe more
detection maybe more
actual code to detect for any of these
things and then the use cases are really
numerous here i'm trying to simplify it
down to some something simple that we
can digest here but
this allows me to have some things that
can scale out to meet demand without
having to
scale the whole application up so i
immediately get that piece here
and so from a perspective of processing
of data
this is great for me and i can also
evolve other sections of my code to work
better scale faster
i want to swap in out different
components all right
even in my decoupling
whereas before monolithically i probably
am writing everything in one language
because it's all one single code base
decoupling gives me the ability to maybe
lean on other languages for certain
expertise i may have my front end and a
java but my backend may be in a node.js
or any other language python i can take
advantage of
many different other benefits that the
languages can offer because again it's
all about the single responsibility of
work that they must achieve here so
to summarize it all message queuing it's
definitely for the new developer
it is something to consider uh from an
architectural perspective of what you
want to achieve for our existing
developers i started out as a lamp and a
ming developer myself started out from
monolithically moved into http
communication between services and this
is definitely a technique that i have
embraced uh in my hybrid architecture to
say there are some circumstances when
these patterns resonate with me i'm like
yes that would fit perfect to work
definitely if you're getting in the
cloud native
being that we have things that are
decoupled um
or at the part that you need many
different pieces to function together to
connect them together message cueing is
a great
segue to get you immediately there in
the cloud native world so definitely a
great technique to try here
hope this video was useful i can't wait
to hear how you gonna implement message
queuing in your architecture thank you
for your time
if you have questions please drop us a
line below
and if you want to see more videos like
this in the future
please like and subscribe