Redelivery count for messages

[hairyhum]

From the 0-9-1 spec:

The server SHOULD track the number of times a message has been delivered to clients and
when a message is redelivered a certain number of times - e.g. 5 times -
without being acknowledged, the server SHOULD consider the message
to be unprocessable (possibly causing client applications to abort), and [dead letter] the message.

This should be made optional for compatibility.

More context is provided in #1020.
Related: https://github.com/rabbitmq/rabbitmq-stomp/issues/36.

Update:

• Quorum queues in 3.8.0 track redelivery counts and expose the counter to clients via a header
• Poison handling in quorum queues #1889 introduces a redelivery limit (poison message handling) on top of that
• Implementation for classic queues is under consideration

[michaelklishin]

This is not really a requirement, and message properties don't have a field for redelivery count, only a boolean flag. Still worth having, of course.

[hairyhum]

First investigation results:
Required changes in backing_queue API to support integer redelivery count instead of flag in fetch/2 return value and functions working with fetch.
Also queue index format should be changed to support multiple deliver records, or delivery count. But because current implementation assumes exactly 1 deliver for each acknowledge there should be significant changes in index structure.
Any thoughts?

[vgkiziakis]

Also very interested in this feature. It's invaluable in scenarios where there is a "poisonous" message that for some reason is causing consumers to crash or behave erratically.

Are there any plans for adding this in a future release ?

[hairyhum]

It's in the roadmap for 3.7.x

[mikljohansson]

@hairyhum do you know if the delivery-count feature is still slated for inclusion in rabbitmq-server 3.7?

The reason I'm asking is because we see the poisonous-message problem every once in a while. Latest incident was last week and caused a system degradation in our own components, before the problem could be found and circumvented.

We'd very much like a robust broker-supported way of detecting messages that consistently trigger a severe problem in the client

[michaelklishin]

@mikljohansson we will do our best to fit it into 3.7.0 but no promises.

[plechev]

This is a much needed feature to enhance the robustness of message (re)delivery mechanism. We are dealing with the "poisonous messages" problem which is currently handled by the client, but it will be more elegant if we can use the broker for this.
Thumbs up!

[michaelklishin]

An update: this won't be in 3.7.0 because we had quite a few out-of-cycle and new features in 3.7.0 and we need to limit the scope at some point. But we are contemplating it for 3.8.0 (so that it doesn't have to wait for 4.0, that is).

[mattheworiordan]

An update: this won't be in 3.7.0 because we had quite a few out-of-cycle and new features in 3.7.0 and we need to limit the scope at some point. But we are contemplating it for 3.8.0

@michaelklishin out of interest, how do your minor release cycles work i.e. realistically if it made it into 3.8.0, when could we expect that release?

Separately, is there anything we could do to help if it was a priority for us?

I posted this issue here as a suggestion for the team to work on it. Your product is really great and I think it would become bigger if retry capabilities are present. Thank you!