Serializing metadata and domain messages

Once upon a time you’re getting a task where you need to deal with data used in the domain process and some which are irrelevant for it. First approach is use them in the main model of event or request but there is a better way.

Creating an event

Let’s take a look on how to serialise data in the language agnostic way for the event bus in our functional system.

Of course we need payload of our event which holds all the important data about what happened in our system. Let’s then serialise it to JSON.

 

Can you tell me what just happened in the system? I don’t think so and the same will apply to any other system.

What we need is a description of the payload we’ve just seen. One important thing we’d need to figure out how to deal with this payload is event name.

Question is – where to put it?

Step 1 – Adding a field

It’s the most obvious way and the worst in my opinion.

 

The one good thing about it is that it’s simple and requires the least effort at the start. It can give you quick win when you prototyping or changing legacy system.

On the downside is mix of infrastructure and domain data in your payload and strange behaviour where you need to read field first and then you’ll know how to handle the rest of the payload. Another is possible name collision (can be avoided by prefixing with “_” etc) and difficulty in extending by new fields.

Step 2 – Extracting name outside of the payload

Let’s take a look how can we improve the event payload so we separate infrastructure and domain related data.

 

As you can see we’ve made some improvement as the event name is not a part of payload of the event and they live on different paths. Deserialization is a bit easier as after reading just one field we can dum the “payload” into our favourite tool to transform JSON to object in our favourite language.  So far so good.

The only problem we can encounter on our way is when we’ll need to add more of metadata as the root of our message will grew and it may be hard to model when not always all the data are needed.

Step 3 – The header way

Let’s step back for a moment and let’s take a look at HTTP protocol. each message, being just a text, contains a lot more than just payload.

 

First line defines protocol version and status of the response. Then we can see headers of the http request containing all the informations needed for the client and not being part of the content to be consumed. At the end we have content of the response which is separated from headers by empty line.

Back to our system and event bus we can take similar approach and define our event as headers and payload. Headers will contain all the information needed for infrastructure of our operations where payload will contain only the domain data.

 

As you can see, by separation you can add a lot of information around your business logic keeping separation of concerns in check. I’ve used HTTP-like headers in my example as they are quite common knowledge and they should be easy to understand for everyone who work with web technologies.

Headers implementation details

In my example I’ve used headers as a part of JSON document sent through the bus. Depends on what you will use you can use this solution or use native message headers.

Kafka with all the features, and with how amazing it is, is solely focused on payload of message. There is no feature of headers, so you have no other choice than embed header in the message itself as I did.

Other story is with AMQP queues, like RabbitMQ, where you can use AMQP message structure and separate headers on the bus level. Similar story when you use Spring Integration framework where you can use headers on all the messages and pass additional information alongside the data.

Conclusion

Messaging is not only about payload of the message. Metadata, here called as headers, are very important part of the communication. Maybe they are not crucial for your domain code but I’d say they are required for your systems to work properly at scale. Separating them will give you a lot freedom and ease of change and complexity of a bit trickier deserialization is small price to pay for the value you’ll get.

CQRS explained – slides

Yesterday I gave a talk about basics of CQS and CQRS. It was a part of Code Mastery meetup I’m co-organising.

Here are my slides. Video should come up soon as well.