Busting event-driven myths

If you think an event-driven architecture isn’t for you, think again: you might be missing out on useful design patterns that will future-proof your applications.

In the past few years, there has been an increase in popularity of event-driven architectures: more conference talks, blog posts, Slack discussions, and tweets on the subject. While event-driven architectures are not a new concept, they are especially well suited to recent trends in software engineering, including cloud computing, microservices, and scaling by building flexible, distributed systems, which explains why they are becoming more popular.

Event-driven architecture is the idea that one should design software starting with events, things that happen in the real-world and have real business meaning, and that these events should be the central concept in the architecture. Data is front-and-center, and components create events, handle events and send events to each other. There are several software design patterns that are particularly suited for event-driven architectures, such as event notification, event-carried state, and event sourcing.

But just like any architecture pattern that suddenly becomes popular, there are a few common misconceptions about it. These misconceptions may even cause some software engineers and architects to avoid event-driven architectures, even when they are a good solution to their problem.

Here are a few popular myths and how they compare to reality.

Myth no. 1: Event-driven architectures are implemented with event-driven programming

As event-driven architectures are gaining popularity, so is the event-driven programming model. This causes quite a bit of confusion, because the two concepts are only loosely related. Both involve writing applications that react to events and changes in the state of the world as their core paradigm; however, the goals of building a system in such a manner are different, and therefore the implementations are not related.

Event-driven programming frameworks are typically used to provide multi-tasking capabilities in languages or environments that lack an explicit threading model, such as Node.js. They provide higher efficiency and performance by using a small thread pool to serve a large number of requests in libraries such as NIO or to provide a nice interface for pluggable frameworks such as Kafka Connect. You can use those event-driven languages and frameworks even if you choose not to implement an event-driven architecture.

Event-driven architectures are all about keeping an eye on the future extensibility of a system with multiple services. By publishing events about changes that happen, rather than explicit commands, it’s easy to evolve the system by adding new reactions to those events or publishing new events that services can react to. Consider a system that requires compressing files. You can have a controller issue a “compress file” command to a service, or you can send a notification that a new file exists, which means it may get compressed, and as the system evolves, it may also get validated or copied to a backup system.

The central patterns of event-driven architectures that help implement this kind of extensibility can be and are implemented in any language, and with many different frameworks.

Myth no. 2: Event-driven architectures must be asynchronous

There are many benefits to asynchronous (async) event-driven architectures, but it is by no means mandatory and there are benefits to an event-driven architecture even if your services are synchronous (sync). Some software developers shy away from using event-driven architectures because they think an event-driven architecture is always async. Because they are not used to writing async services, they decide to avoid the whole architecture, missing out on its benefits.

In the file compression example, an async “new files” service sends the new file notification to the event bus, and there is a good chance that it will be done at that point. If it does have a reason to know that the file was successfully compressed, it will subscribe to a topic with “file was compressed” events and react to these events. A sync “new files” service will send a new file notification to the event bus and wait for a response from the compression service saying that the file was compressed before going back to its other duties.

You can see how the sync version can be simpler to write and to understand, while the async version will scale better as the number of files grow. Both are event-driven services because both communicate by publishing and subscribing to events that describe changes in the state of the world.

Myth no. 3: Event-driven architectures are inherently more complex

A flexible and loosely coupled architecture in which independent services respond to events has advantages in scalability and agility, but it does come at a cost. Because there is no central controller that manages all communication flows, tracking down issues in event handling or even understanding the entire system involves more moving parts. This is of course by design. If you’re building a microservices application, you typically don’t want one big controlling service sending orders since this architecture is so challenging to extend.

However, there are several things you can do to make the system simpler while maintaining agility:

1. Most important, manage shared data well. The usual options are either to use a shared database or to give every service its own database, but both have well-known pitfalls. Make sure you are aware of the challenges and solutions in this space; this guide to event-driven services is a good place to start.
2. It is also important to have strong monitoring and observability solutions in place. You need to know which service sent which events, and who is subscribed to these events. Having good visibility into the flow of events will let you understand the system and troubleshoot it with more confidence and less guessing.
3. Be very mindful of the composition of services and their interfaces. With all the buzz about microservices, it is easy to create something too complex by breaking down the system into too many services. One way to approach the design is to model your event flow first. List the events that make up your application: things that actually mean something in the real world. A good technique to get this process right early on is event storming. Then, you can use this list of events to guide what services you need to respond to them. If your business user doesn’t understand what your service does simply from its name, you’ve probably gone too far.
4. Remember that the schema of the events published is now the API between services. You need to make sure all consumers can easily understand the events in topics they subscribe to and that all changes to events are done in a way that maintains compatibility and does not cause downstream services to break.

Myth no. 4: Event-driven architectures can be implemented without a central event bus

Some teams attempt to implement an event-driven architecture using REST APIs to communicate between services. This may seem reasonable at first: Why not use an API that I’m already familiar with to send events to other systems? But if you agree that one of the main goals of event-driven architectures is to help you create a more extensible system, you’ll see that using direct point-to-point communication between services does not help in achieving this important goal. The resulting architecture is a services architecture, but not an event-driven one.

Having a central publish/subscribe system to which any service can send events, and any service can subscribe to a topic of events that it is interested in, is key for extensibility. In my example above: if the “file upload service” directly notified the “compress file” service about new files, it would be significantly more challenging to add the new verification and backup services. If the “file upload service” publishes events about new files to a central topic, I can add as many services that handle new files as I need, completely independently. This is a more agile and less coupled architecture.

A modern event-driven platform adds three important benefits:

1. It scales extremely well to large number of producers, consumers and messages, so you never need to worry that by adding your new service you’ll cause the entire architecture to tip over.
2. It can store events for long periods of time without degradation in performance, potentially forever. This helps implement the Event Sourcing pattern that is often used in event driven architectures.
3. It provides a central schema repository, allowing new services to understand the events stored in each topic, subscribe to correct topics and extend the events in a compatible way.

Event-driven architectures provide significant agility and scalability. I’ve seen many engineering groups use them to increase their velocity, productivity and ability to implement challenging and yet valuable business requirements. Don’t let a few old myths scare you away from using this valuable tool.