Serializing metadata and domain messages

Once upon a time you’re getting a task where you need to deal with data used in the domain process and some which are irrelevant for it. First approach is use them in the main model of event or request but there is a better way.

Creating an event

Let’s take a look on how to serialise data in the language agnostic way for the event bus in our functional system.

Of course we need payload of our event which holds all the important data about what happened in our system. Let’s then serialise it to JSON.

 

Can you tell me what just happened in the system? I don’t think so and the same will apply to any other system.

What we need is a description of the payload we’ve just seen. One important thing we’d need to figure out how to deal with this payload is event name.

Question is – where to put it?

Step 1 – Adding a field

It’s the most obvious way and the worst in my opinion.

 

The one good thing about it is that it’s simple and requires the least effort at the start. It can give you quick win when you prototyping or changing legacy system.

On the downside is mix of infrastructure and domain data in your payload and strange behaviour where you need to read field first and then you’ll know how to handle the rest of the payload. Another is possible name collision (can be avoided by prefixing with “_” etc) and difficulty in extending by new fields.

Step 2 – Extracting name outside of the payload

Let’s take a look how can we improve the event payload so we separate infrastructure and domain related data.

 

As you can see we’ve made some improvement as the event name is not a part of payload of the event and they live on different paths. Deserialization is a bit easier as after reading just one field we can dum the “payload” into our favourite tool to transform JSON to object in our favourite language.  So far so good.

The only problem we can encounter on our way is when we’ll need to add more of metadata as the root of our message will grew and it may be hard to model when not always all the data are needed.

Step 3 – The header way

Let’s step back for a moment and let’s take a look at HTTP protocol. each message, being just a text, contains a lot more than just payload.

 

First line defines protocol version and status of the response. Then we can see headers of the http request containing all the informations needed for the client and not being part of the content to be consumed. At the end we have content of the response which is separated from headers by empty line.

Back to our system and event bus we can take similar approach and define our event as headers and payload. Headers will contain all the information needed for infrastructure of our operations where payload will contain only the domain data.

 

As you can see, by separation you can add a lot of information around your business logic keeping separation of concerns in check. I’ve used HTTP-like headers in my example as they are quite common knowledge and they should be easy to understand for everyone who work with web technologies.

Headers implementation details

In my example I’ve used headers as a part of JSON document sent through the bus. Depends on what you will use you can use this solution or use native message headers.

Kafka with all the features, and with how amazing it is, is solely focused on payload of message. There is no feature of headers, so you have no other choice than embed header in the message itself as I did.

Other story is with AMQP queues, like RabbitMQ, where you can use AMQP message structure and separate headers on the bus level. Similar story when you use Spring Integration framework where you can use headers on all the messages and pass additional information alongside the data.

Conclusion

Messaging is not only about payload of the message. Metadata, here called as headers, are very important part of the communication. Maybe they are not crucial for your domain code but I’d say they are required for your systems to work properly at scale. Separating them will give you a lot freedom and ease of change and complexity of a bit trickier deserialization is small price to pay for the value you’ll get.

Event driven microservices

I’ve got lucky I got my spot in last London GOTO conference. Theme was obviously event driven microservices and how they simplify whole architecture of complex applications. After 2 days of listening about them there was no other choice than prototype.

Most intriguing for me was the idea of performing read through event bus as well as writes. It kinda hit me as being against CQS. Couldn’t get why and how. Being in mindset of transactional storage wasn’t helping as well :P

Building event driven microservices

Requirement as simple. One service is providing data and other is requesting and processing it. And some interface to interact with. No need for web, as it’s prototype/proof of concept. Console app it is and Symofny as framework of choice as the simplest way to scaffold code. Yesterday I was thinking about getting kafka but again, I wanted to see results not caring much about technology.

If you want to do system like that in real life please don’t use PHP for that :D

Code you will find here.

So how it works?

I’ve got positively surprised by the effect. Everything was obviously done under a second. Even checking on microsecond level shown loads of time saved in comparison to HTTP.

Think this way – sending order and getting item gives you 2 HTTP calls. It’s not much but with more services interested in this topic it will only grow. And all of it will be done during main order request. So user facing system will be slowed down by every new service attached. Even if response time is about 50ms with only 4 of them it adds 200ms on top of all other processing.

When your microservices communicates with each other with events you’re getting at least three advantages I can think of right now:

  1. No HTTP, less network connections means less time wasted on the wire
  2. Events are asynchronous so if your critical path is finished you can inform end user about it without waiting for all the rest to finish
  3. If one of services in chain fails events will stay in the infrastructure. You’re not loosing any data.  Rest of system is working as normal. Error is atomic to one place. And after it’s fixed you can start processing messages from where it stopped.

Last point is illustrated in my example. When you request non existent item ItemService will fail. Fix collection adding new index and rerun it. Boom. Order gets processed like nothing happened. Just with a bit of delay.

Conclusion

If you can use this pattern it will give you a lot of freedom. Communicating with event bus of any sort will save you a lot of time and money. One thing to do is just to convince your team and company it’s the right thing to do. But it’s topic for another post ;)