An event driven architecture, Part 4

Frameworks, tools and conclusion

Mini frameworks for developers

To ease the development of EDA services we created a dedicated framework, Eventail, that abstracts away the topology and all required behaviors of the services as described in part 3 so that the developers can focus on the business logic of their services. They just need to implement callbacks for each event or command they need to process.

We released it under a MIT license for Python, Elixir and Rust:

• Eventail-py
• Eventail-ex
• Eventail-rs

Monitoring

Beside the “classical” network and server monitoring solutions, we use more high level, specific tools for monitoring our event driven architecture.

As global bus health monitoring, we use the RabbitMQ management plugin. It provides a live view of what’s happening on the bus:

• message throughput;
• queue states;
• broker resource usage;
• client connection states…

As we said earlier, the services send their application logs on the bus, on a dedicated exchange. The Eventail framework formats the log messages in the GELF format used by Graylog so we can leverage that log management solution which can natively connect to AMQP brokers like RabbitMQ. We then use Graylog to define alerts, create graphical dashboards on the business processes, search the logs for debugging (by conversation identifier for example), etc…

In addition to application logs, the services also send periodical health checks on the same exchange, and those are also collected in Graylog: at any moment we know which services or workers are down, or are running in a degraded state.

Testing, debugging and “repair”

The Eventail framework provides command line utilities for testing, debugging and replaying messages in the dead letter queue.

The testing commands allow the developer to publish events on the bus, or to send a command to a service and display the result on the console.

The debugging commands allow the developer to target and display logs on the console, or to subscribe to events and commands in order to display their content as they occur. There is also a command to inspect the content of a queue (without consuming its messages).

Dead letters

We use RabbitMQ policies for that.
Services just have to refuse a message and flag it in order to let the broker route it to the dead letter exchange. The framework manages this automatically when an unexpected exception is raised in the user code.

There should be something listening to the dead letters though, for this device to be useful.
As we saw, if a bug happens in a service, the message that raised the error is sent to the dead letter queue. There we can inspect its content with the inspection tool. We can also save it in order to “play” it on the developer computer with the publish_event or send_command utilities. Once the bug is fixed in the service and is deployed in production, we can replay all the messages in the dead letter queue with the resurrect command. Thus, no data is lost.

Outcome

What do we get?

After six years in production, our event driven architecture brought us:

• Scaling: just start a new instance of a logical service, anywhere on the network, to keep up with the charge.
• Peace of mind: no data is lost, ever. No more stress. No more panic when something goes wrong in production.
• Lower latency: we got rid of polling processes, cron jobs, batches…
• Streamlined monitoring and debugging.
• Trouble free maintenance: thanks to decoupling we can add, stop, restart or retire services as we want, without disturbing the system.
• Painless release of new features: just add a new service, it’s completely transparent to others.