Are you overselling EDA?

This statement inherently makes a comparison, usually between EDA and the now traditional REST API method of connecting components. It states that EDA is more loosely coupled from a functional, temporal, and location perspective. Diagrams as shown below are often used to illustrate this.

Let’s examine this.

The argument is that in an EDA, the event producer does not know who will eventually consume (use) the event he produces. This is certainly true. However, I believe this also applies to many REST APIs out there. For instance, Google likely did not anticipate that my local bakery would use its Maps API to calculate delivery costs. Moreover, with the introduction of API gateways and service discovery, the providers and consumers of APIs are not “tightly” coupled either. This doesn’t mean that Google ignored their needs during the implementation of their REST API. Instead, it indicates that they applied a similar level of strategic thinking to what an event producer would consider when designing the structure of the events they intend to publish.

By the way, the drawing mentioned above can also be presented like this:

As event producers and consumers are decoupled through the broker, they can function independently. Events can be produced without a consumer being present, and events can be consumed -if they have been previously published- without the producer being active. This is less obvious with REST APIs, where the endpoint must always be accessible to the client.

But let’s dig a little deeper. This ‘advantage’ is linked to the asynchronous nature of the communication channel. Events are by nature asynchronous, whereas REST APIs are not. As an implementer ,you have the choice to make them either synchronous or asynchronous. Most of the REST APIs we encounter are synchronous (because it’s easier to implement them this way) but they do not have to be. If you want this benefit, you have to code it accordingly.

Side note: Asynchronicity is not always a blessing. In this paradigm, not having a provider/consumer is considered “normal”-it is not really an exception. However, this can be problematic. For instance, consider the situation where an event producer hasn’t generated “events” for a number of hours. These types of issues are very visible in a synchronous world; however, in an asynchronous world, you will need to create logic to detect these kinds of problems.

Location Independence: Sometimes, it is argued that event systems are more location independent because they are capable of receiving messages from various operating locations, both on-premises and in the cloud. This is certainly true, but it is also the case for API gateways that allow you to expose APIs located either on-premises or in the cloud.

Although many people believe this, I do not agree with this statement. When defining an event, both its structure and content, one should always consider the purpose of the event (see my blog on event types). The purpose (use case) of the event and the specific requirements of your organization or business context are crucial in designing the correct event message. The requirements for events geared toward data sharing differ significantly from those aimed at process automation. Failing to acknowledge these differences can lead to a suboptimal and potentially complex design, which may hinder the future evolution of your architecture.

This is also a misconception: scaling, even in an event-driven world, does not come for free. It requires thorough analysis and a deep understanding of how the system must be able to scale.

Most of us are familiar with vertical scaling, where you increase the size of your machines as your workload grows. This involves adding more CPUs, memory etc. However, this type of scaling has its limitations: the number of CPUs and memory in any given machine is limited. On the other hand, vertical scaling can be easily implemented as it does not impose any specific requirements on the program or application. For example, running your favorite game on a new iPhone, which is equipped with a faster CPU, will probably perform better than on your current phone, which is five years old. However, the App developer did not need to modify his program; it simply benefited from the faster CPU.

However, there is a second type of scaling known as horizontal scaling. This method has also been around for a long time, but with the rise of cloud computing, more people have begun to see its benefits. The idea here is that instead of increasing the size of a single machine when the workload increases, we distribute the workload across multiple machines. Theoretically, if you had an infinite number of machines, you could scale without limit. However, there is an important caveat: you must be able to distribute your workload effectively. This is not always possible, or at least it requires careful planning; A famous example of where this approach does not work is the saying, “It takes nine months to make a baby“. Adding more “women” does not decrease this time (nine women will not create a baby in one month).

It is argued that an event-driven system will better support horizontal scaling, but I tend to disagree. The success of scaling depends on the design of the system and the use of asynchronicity.

For example, imagine a company operating a helpdesk that needs to process complaints. This company is active in Belgium and Luxemburg, and is also planning to expand into the Netherlands.

While designing the system, the architects decided to leverage horizontal scaling to accommodate the potential growth of the company. They designed the system in such a way that the workload could be split by country.

The front-end application will call a “Callback” REST API on the API Gateway. The gateway will first buffer the requests (by country). It will then read the requests from the buffer and call an API in the back-end to process them. If the back-end is not available, the requests stay in the buffer until it becomes available again. The back-end is constructed with a (micro)service architecture and is deployed in a scalable (cloud) infrastructure. This means that when one of the back-end servers becomes overburdened, a new server spins up to handle the extra load.

This results in a perfect scalable and resilient system. Requests are managed by an API gateway that can scale to accommodate more requests. Additionally, the back-end implementation can be scaled. If we expand into the Netherlands, we simply configure a new scalable back-end server, and, if needed, add a processing node to the API Gateway cluster.

The front-end application would post the request as an event on the broker, with the broker maintaining a queue for each country. These country-specific queues would be monitored by back-end services, which can also scale depending on the number of unprocessed messages in the queue. When expanding to the Netherlands, we simply define a new queue and the corresponding back-end listeners.

From a scalability and resilience point of view, both solutions are equivalent. Other factors (not mentioned in this simple example) will determine which implementation solution is better.