Outbox Pattern — Avoid Duplicate Messages

You'll already be using a message broker that supports at least one delivery. At least once delivery guarantees that a message will be delivered once or more that you have the same message that could be delivered to a consumer possibly multiple times.

Duplicate messages could have some very negative impacts on your system. For example, an order being places twice.

In this article, we are going to understand why do duplicate messages occur and how do we deal with them.

Let's start talking about delivery guarantees.

• At Most Once
• At Least Once
• Exactly Once

Message brokers don't all support this, they generally support probably one or two but they are at most once and this means is that a message will get delivered to a consumer maybe once. This means that it might not at all or most it will get delivered once.

There's also at least once and probably what you're going to pick a message broker that supports and that means again that a message will get delivered guaranteed at least once but it could get delivered more than once.

So, at least once delivery generally works is that your message broker will send a message to your consumer and then your application code basically consumes that message and send an acknowledgment back to the message broker.

At this point, our message broker knows that message was processed and it won't deliver to another consumer. So, why do duplicates occur?

For one is that if we deliver our message to the consumer but you completely forget to send back the acknowledgment, generally what the brokers have is a timeout. They expect an acknowledgment back after a certain period of time. If you've never sent it then they're going to resend that message to a consumer and if you've already processed it, now you have a duplicate.

The same scenario is because there is that timeout maybe you do send the acknowledgment but you don't do it within that time frame that the message broker expects. So, again it could send that message before you even send the acknowledgment.

So, another reason why this might occur is that if you are using the outbox pattern as I posted in my previous article where you have your producer publishing messages not immediately to the message broker but rather a database.

The producer fetches those records from the database and then publishes those to your message broker. It needs to go update the database or pull those messages from saying that either deleting the records or marking them as being published.

What happens when it can't go back and update those records. It's going to republish those same messages to the message broker. So, there's another reason why you could have duplicates.

For each consumer, we're going to create a new table in our database that lives alongside these other tables.

I've created this IdempotentConsumer that's going to be an entity that has the message-id and the consumer and then in my database the message-id and the consumer are actually going to be the primary key. So, that when you actually commit that transaction we're only ever going to be able to do that if that combination of message-id and consumer does not exist yet in that table because if it does it means that we've already processed that message.

In the first method, we're just adding a new record for the item messages that I've processed. Then I've added another method called is processed where we're just taking the same message-id and the consumer and checking that to see if that already exists within our database. So, we’re going to use these two methods to handle duplicates.

We'll pass the message-id and then the reason why this is done is by the consumer is because that you may have events that are processed by different actual handlers in different areas. You want to be doing this deduplication per actual handler. So, if we have processed this then it can just return and that’s the end of it. We don't need to process it again.

Now, we actually need to save that we've actually processed this message and so that’s how you deal with duplicate messages.

Have a nice reading!