Message Brokers: Give you app an unlimited power

7 min readFeb 7, 2023

In traditional model for communication between two services, we need to send the data over the network in packets(bytes). We send the data from our machine in some format to another machine that understand that format. There might be a case that the receiver or server is not available (or down) , then there is loss of that request data. what is the solution for this?

There are two side of it, client and server.

Client is what send the query to request some data and server is the part that handle the request and responds with result.

Message brokers works differently, you send the data but it works asynchronously and it doesn't wait for the response. You messages are stored in the broker so there is no risk of data loss as well.

In this blog, let’s try to understand what is the idea behind message brokers and how they are different from other modes of communication.

What are message Brokers?

Message brokers are application systems that allow you to communicate with applications asynchronously, and helps you to create services that are decoupled.

Message Brokers plays an important role in scaling large systems. Every big company uses brokers to make their system scalable and reliable.

Message brokers store data in buffer so that if there is no one at the moment to receive the data , data should not be lost.

A message broker is a software that acts as an intermediary for exchanging messages between applications or systems

Still not clear about message brokers and why you need them? Let’s try to understand in simple ways

Let’s say you have an application that allow users to upload videos. There are multiple things going behind the scene. There are some processes that takes longer time and you don’t want to user to wait for these things to get time, which is really a bad experience. So what is the solution to this.

The first thing that comes to your mind is the decoupling of services and making the services asynchronous, so that they can work without the user waiting for them to get done.

We introduce the concept of message broker in this.

Once a video has been updated to the server, After validations and security checks are done, we can push the event to the message broker and return the user with response. From the message broker, we have defined topics(topics are like a type in which you can push messages and receiver and get messages from that type) and messages are pushed to broker in form of bytes and stored in message broker.

Once There is a receiver available, they can request data from the broker and process the request, there is no data loss as messages are stored in broker.

Brokers are highly scalable, durable and available.

Message brokers can be used in a variety of situations where applications or systems need to exchange messages with each other, and can provide a centralized point of control for message exchange, allowing for greater scalability, reliability, and security in communication between applications and systems.

The message broker is responsible for receiving messages from the sender, storing them in a queue, and delivering them to the receiver

A typical architecture of a message broker includes the following components:

Producers — Producers are the applications or systems that produce messages and send them to the message broker for delivery.
Message Broker — The message broker is the central component of the architecture, responsible for receiving messages from producers, storing them, and forwarding them to subscribers.
Message Queue — A message queue is a data structure used by the message broker to store messages until they can be delivered to subscribers. In some message brokers, multiple queues can be used to separate messages by topic or priority.
Subscribers — Subscribers are the applications or systems that receive messages from the message broker. They can receive messages in a variety of formats, including publish/subscribe or queue-based delivery.
Routing — The routing component of the message broker is responsible for forwarding messages from producers to subscribers based on the rules specified by the broker configuration.
Security — Message brokers typically include security features to ensure that messages are protected as they are being transmitted and stored. This can include encryption, authentication, and access control.
Monitoring and Management — Message brokers typically include monitoring and management features to help administrators keep track of the performance and behavior of the broker, and to diagnose and resolve issues as they arise.

Message brokers support a wide range of messaging protocols, such as:

Advanced Message Queuing Protocol (AMQP) — AMQP is a standard for messaging middleware that allows applications to communicate with each other by exchanging messages. It provides features such as reliability, security, and transaction management, making it a popular choice for enterprise-level messaging solutions.
Simple (or Streaming) Text Oriented Messaging Protocol (STOMP) — STOMP is a simple, text-based messaging protocol that is easy to implement and use. It provides a basic set of messaging features, such as publish/subscribe and message queueing, making it suitable for a wide range of applications.
MQ Telemetry Transport (MQTT) — MQTT is a lightweight messaging protocol designed specifically for IoT devices and other resource-constrained environments. It uses a publish/subscribe model and is designed to handle large numbers of connected devices, making it a popular choice for IoT applications.

Event driven architecture uses message brokers for decoupling them between services.

There are scenarios where the load on system is high and you want things to process really fast or even at same time (parallel). Then comes the concept of Partitions and Groups.

Partitions: Partitions refer to the logical divisions of a queue within a message broker. They allow for the queue to be divided into multiple parts, each of which can be processed independently. This can help to scale the system, as messages can be distributed across multiple partitions for parallel processing.
Groups: Groups are a way of organizing consumers within a message broker. A group of consumers can be assigned to a specific partition, allowing for messages to be processed in parallel by multiple consumers. This can help to improve the performance of the system and reduce the load on individual consumers.

Let’s try to understand how partition and Groups plays an important role in scalability.

Imagine a financial company that processes a large volume of transactions every day. The transactions need to be processed quickly and accurately, and the company wants to ensure that no transactions are lost or duplicated.

To handle the processing of transactions, the company can use a message broker that supports partitions and groups. The transactions can be divided into multiple partitions, and multiple consumers can be organized into groups to process the transactions.

Here’s how the architecture could work:

The company divides the transaction messages into multiple partitions within the message broker. For example, the transactions could be divided by region or by type of transaction.
The company creates multiple consumer groups, each with several consumers. Each group is assigned to process messages from one or more partitions.
The consumers within each group process the transactions and update the company’s financial systems, such as updating account balances and generating reports.

By using partitions and groups in this way, the company can balance the load of transaction processing, ensuring that transactions are processed quickly and accurately. If a consumer fails, the messages in the partitions assigned to that consumer will be automatically redirected to the other consumers in the same group, ensuring that all transactions are processed in a reliable manner.

I hope this gives you an idea about message brokers. There are many message brokers available, ranging from open source to commercial solutions. Some of the most popular message brokers are:

Apache Kafka: An open-source, distributed, scalable, and high-performance message broker that supports publish/subscribe messaging, as well as storage and processing of streams of records.
RabbitMQ: An open-source message broker that supports multiple messaging protocols, including AMQP, MQTT, and STOMP. It is known for its ease of use, reliability, and scalability.
ActiveMQ: An open-source message broker that supports multiple messaging protocols, including AMQP, MQTT, and STOMP. It is a popular choice for enterprise messaging due to its support for transactions and its ability to integrate with other systems.
Microsoft Azure Service Bus: A commercial message broker offered by Microsoft as part of their Azure cloud platform. It supports multiple messaging protocols, including AMQP and MQTT, and provides a managed, scalable, and secure messaging solution for cloud-based applications.
Amazon Simple Queue Service (SQS): A commercial message broker offered by Amazon Web Services as part of their cloud platform. It is a managed, scalable, and highly available message queuing service that supports a variety of use cases, including decoupled microservices, batch processing, and fan-out broadcasts.

In coming blogs, I’ll try to explain these message broker with working example. Stay tuned !

If you liked this blog and would like to read more about this, stay tuned and subscribed for email notifications whenever i post new blog.

Do checkout these blogs:

Other Architecture blogs:

Other Life changing blogs for productivity and Focus: