Welcome to today’s session where we’ll explore how to implement the Enterprise Service Bus (ESB) pattern using Azure. We’ll dive into the key components, tools, and best practices needed to effectively handle message routing, orchestration, and transformation across various systems. Whether you’re looking to manage synchronous and asynchronous processing or ensure reliable message delivery, this video will guide you through the essentials of building a robust ESB on Azure. Let’s get started!
Today, we’re going to talk about what an ESB might look like on Azure. In the last video, we discussed the benefits of using the ESB pattern, and now we’re going to delve a bit deeper into what that actually means and how we can implement it. Let’s see if we can cover this in two minutes.
To support an ESB pattern, you need to handle both synchronous and asynchronous processing. You must be able to route messages between different systems and destinations, accommodate various communication protocols, and orchestrate processes in a sequence. Additionally, you need to handle events—if something happens, then something else should follow. Reliable delivery is also essential; we need to be confident that messages placed in the ESB will reach their intended destination. Management is critical too—we need to monitor, support, and understand what has happened and identify any issues.
We also have to transform messages. Not only might different systems use different communication protocols, but they may also use different schemas. So, we need to be able to transform one message schema to another—not just from XML to XML, but from JSON to CSV, and so on. Validation, which again involves schema, is also necessary. To validate a message, you first need to define its format—its schema—and then validate against it. We need to support all these functions and more, but these are the key ones.
We also need to consider the patterns we can implement. A crucial one is publish and subscribe. One system needs to inform the world that something has happened, and other systems can then subscribe to that event. Another important pattern is batching and splitting. For example, one system might output a large file, but the next system can only process one item at a time, so we need to batch and split—or de-batch—accordingly. Additionally, we have scatter and gather, where we retrieve information from multiple systems and consolidate the results into a single piece of work.
We need to be able to handle all these types of message exchanges. For example, fire and forget—sending a message and considering the task done. Ordered delivery—sending messages in a specific sequence, ensuring they arrive in the correct order. Sequencing might also involve sending different types of messages in a specific order. These are just a few examples of the sequencing and message exchange patterns we must support.
Now, when we look at implementing this in Azure, we have some really rich tooling available. We don’t need too many different Azure services because we have a growing toolkit. One of the key components of an ESB is the on-ramp—this is where information enters the ESB. We typically use Logic Apps for this because they provide numerous connectors that help us connect to other systems and bring in information via different protocols: HTTP, FTP, AS2, etc.
We have a wide range of connectors to help with the on-ramp. We can also use Azure Functions to write custom code and run it serverless. Between these tools, we have most things covered. Moreover, within this setup, we can also handle transformation and validation. Generally, by the time messages reach the messaging engine, they should have been validated and transformed into an intermediate canonical format. The off-ramp is essentially a mirror image of the on-ramp; it’s the same toolset. We perform transformation and then transmission. Logic Apps still have the connectors for sending messages, or we can use Azure Functions for this purpose as well. The same toolset is used throughout.
In the middle, you have your messaging backbone, where you have options depending on your non-functional requirements, such as message size, latency, and throughput. The most common choice is Azure Service Bus, which provides a reliable messaging backbone. It supports various subscriptions for publish-subscribe patterns, offers scalability, sequencing, and many other features essential for a core messaging engine. We also have Event Grid for more lightweight event processing—small messages, high volume—and Event Hub for streaming data that can handle really high throughputs. Depending on your needs—whether it’s handling large or small messages—we can choose the right technology.
On top of that, we have orchestration, the system’s brain. Generally, Logic Apps are our go-to technology for this, as they allow us to design processes as if we were creating a flowchart, taking the process from start to finish through several steps.
So, that’s the general toolset we use for implementing an ESB pattern on Azure. Remember, ESB is not a product; it’s a pattern. But Azure provides a very rich toolset to help us implement that pattern. If you want to learn more about how this pattern works or how to implement it, feel free to reach out to me. I’m happy to help.
