In today’s video, we’ll be exploring a key integration pattern that’s crucial for enterprise systems—the sequential convoy pattern. This concept allows you to manage large data sets by breaking them into smaller chunks, making it easier to transmit and reassemble data efficiently. We’ll discuss when and why you’d use this pattern, as well as some real-world examples like API pagination and time-based transaction logs. Stay tuned to learn how the sequential convoy can optimize your data handling processes!
Today, we’re going to look at another key integration pattern, a concept that is very useful for your ESB, or Enterprise Service Bus, and that is the convoy pattern, more precisely, the sequential convoy pattern. We can discuss parallel convoys another time. The general idea behind the convoy pattern is that if you have a data source and a data set that is quite large, and it’s too big to send in one go, you can split it into smaller pieces and send those chunks over the wire. At the target, you then take all those chunks and reassemble them.
What typically happens is that the first item in the sequence will signal, “Hey, I’m a convoy of data,” and the last one might indicate, “I’m the final message.” However, sometimes you just wait for a certain amount of time and assume you’ve received the last one. In other cases, you may find a piece of data and assume it’s the first one. The point is that by splitting your source data set into chunks and reassembling them at the target, you can reconstruct the data set while only transmitting small chunks that are easier to send.
Protocols like web requests, such as HTTP, don’t handle large chunks of data well. Similarly, when you’re using something like a broker queue, they don’t manage huge chunks of data efficiently. But if you take a large data set, break it down, and use a convoy to reassemble it, you’re able to work around those limitations.
Now, when might you use this approach? An obvious scenario is when you have a very large file that you need to break into smaller pieces for transmission. But there are less obvious examples too. These days, a lot of integrations are done with APIs. For instance, when you interact with a REST API and perform a search, you might only retrieve the first 10 records at a time, then you have to press “Next” to get the next 10, and so on. That API pagination is actually a form of convoy because to get the full data set, you need to assemble all those pages. So, API pagination is a type of convoy.
There are also time-based scenarios. For example, let’s say you have a system that writes out a transaction log every hour. At the end of the day, you want to gather all the transactions from that day. In this case, you’re assembling hour-by-hour transactions into a complete data set for the day. These time-based scenarios also fit the convoy pattern well. It’s fairly straightforward to implement, though we’ll go into the details another time. For now, just remember: large data sets in small chunks, transmitted in sequence—that’s your sequential convoy.
