Home Fitness Training MessageBus – Dead-Letter-Exchange

MessageBus – Dead-Letter-Exchange

11 min read
Comments Off on MessageBus – Dead-Letter-Exchange

Written by Martin Führlinger, Software Engineer Backend


In my earlier posts concerning the message bus, I wrote about using RabbitMQ for decoupling our services and how we defined our message content. The final one was about how we keep our receivers fast and resilient. Now I wish to present extra insights into useless letter dealing with, which was talked about in my earlier posts.

Dead Letters

A message can change into a useless letter for varied causes. As talked about in my earlier weblog submit about keeping receivers fast, invalid content material of a message could cause the message to not be accepted. In this case, the message just isn’t acknowledged (NACKed). But there are a number of different causes which may trigger them to not be acknowledged as effectively, akin to if the enqueuing into the Sidekiq doesn’t work correctly, possibly as a result of the underlying Redis just isn’t out there.

No matter which purpose led to the NACK, the message is pushed again into the message bus. In this case you may cross a parameter, which defines whether or not it will likely be redelivered (default) or not.

def process_message(receiver, channel, delivery_info, payload)
  call_receiver_method(receiver, payload, delivery_info.routing_key)
  ack(channel, delivery_info.delivery_tag)
rescue StandardError => error
  nack(channel, delivery_info.delivery_tag, requeue: requeue?(delivery_info))

If a message is NACKed with the requeue choice set, it’s redelivered instantly. This could cause fairly excessive redelivery charges which may, in the very best case, decelerate the entire RabbitMQ, and even kill it if too many messages are redelivered to too many customers. Our inside implementation of useless letter dealing with retries NACKed messages as soon as. In case of invalid content material, it might clearly be NACKed once more. If there was one more reason, it would work on the second attempt on one other shopper machine. If the processing nonetheless doesn’t work, it’s dropped afterwards.

# If the present message was already redelivered
# as soon as we won't requeue it once more.
def requeue?(delivery_info)

Dead Letter Exchange

Dropped messages are principally misplaced eternally, except you’ve gotten outlined a dead-letter exchange (dlx). This trade is created to gather all useless letters and cross them right into a particular queue, which we named “all.dead-letters”. This queue is created manually, too, and simply “stores” all of the misplaced messages.

To create the useless letter trade we took the next steps:

  1. Create new FANOUT trade with the title “all.dead-letters” and the next choices:
    1. sturdy: true
    2. inside: true
  2. Create new queue “all.dead-letters”
  3. Bind “all.dead-letters” queue to “all.dead-letters” trade
  4. Define the queues to make use of the dead-letter trade. This may be carried out individually for every queue or utilizing a coverage (see dlx documentation).

Cleaning up

Once we arrange the dlx appropriately, all NACKed messages find yourself within the outlined queue. This results in a better quantity of reminiscence/storage used, which must be monitored. Our OPS workforce has some alerts on the dimensions of that queue. As quickly they get alerted, they ensure that the backend workforce cleans up these invalid messages.

To clear up that queue, we wrote a small script. The script connects to the RabbitMQ, attaches to that exact queue, and simply behaves as every other shopper from RabbitMQ’s perspective. As lengthy as this shopper is operating, it receives the messages. It then reads the message content material, and its meta-data and decides what to do. This may be one of many following issues:

  • Requeue to the unique queue, which suggests one other retry.
  • Drop the message, e.g. it has invalid knowledge.
  • Requeue it into the dead-letters queue.

Our metadata incorporates, amongst different issues, the goal queue title. You can use this data to course of messages which failed at queue XY throughout one run of the script, and ignore all of the others. Retrying once more is sensible if the problem was not message associated, like out-of-memory on the customers or an unreachable Redis. Dropping is sensible if the message can’t be learn, so if it has invalid knowledge, or if the patron of the message would simply drop it once more since it’s already outdated. If the message is put again within the dead-letter queue, it’s obtained by the operating script so long as it’s operating.

To resolve what to do, you need to know what occurs within the system and what’s the aim of that exact shopper.

It would even be doable to implement some extra refined options, like routinely retrying or dropping a message after a sure period of time, or storing them in one other database.


To stop infinite redeliveries of invalid messages, messages must be NACKed with out requeuing enabled. To stop related messages from being dropped, they should be collected and reprocessed in some way. Whether the reprocessing is completed routinely or manually and what to do with single messages extremely will depend on the use-case and sort of message, and in addition on the quantity of useless letters collected over time.



Source link

Load More Related Articles
Load More By Neil Johnson
Load More In Fitness Training
Comments are closed.

Check Also

The Best Free Bedtime Yoga Videos on Youtube • Yoga Basics

Yoga is an effective way to unwind and chill out earlier than mattress, however typically …