Seeing "outbound buffer limit exceeded" errors from Nats streaming.

[fowlerp-qlik]

Hi

We are running nats and nats streaming as Kubernetes micro services. When a new version of these are deployed we are seeing 1000s of "outbound buffer limit exceeded" messaging being reported by one of the nats streaming pods (not always the the same one).

My question is. Does this error mean that messages are being lost by nats streaming?

[scottf]

@fowlerp-qlik Are you referring to "STAN", the old nats streaming or are you referring to JetStream?

[fowlerp-qlik]

I am referring to nats streaming.

[scottf]

@fowlerp-qlik Transferred to stan.java discussions. @kozlovic Any idea on this?

[kozlovic]

NATS Streaming server does not have the ability to detect duplicate published messages like JetStream does. At the time, the decision was made to make the libraries set the reconnect buffer to very low size (or disabled if possible) to make sure that no published message while disconnected would be accepted. Why? Suppose the following sequences if there is a reconnect buffer.

• Client gets disconnected
• User does a STAN.Publish(), which is added to the outbound buffer, but since the library is not connected, does not reach the server
• Since a publish requires an ACK from the server, the Publish() call will "timeout". The user may think that the publish did not make it and send again. Now 2 messages are in the outbound buffer, and this process could repeat.
• The NATS connection reconnects and library flushes outbound buffer, causing those messages to be duplicated.

That being said, what is described above could happen without a disconnect. It is always possible that a client would get a timeout because of a small value and server being overloaded - but still had received the message. But the duplicate-on-reconnect situation was something that we wanted to eliminate. Of course, if the user provides their own NATS connection, then the STAN library does not have control of that.

And to answer the question about "messages are being lost by nats streaming": not by NATS Streaming, but by the application if the Publish() error is ignored and not retried later. With this error, which is different from a timeout, there is certainty that the message did not reach the server at all. With timeout, it is difficult to know.

[fowlerp-qlik]

In our case the "outbound buffer limit exceeded" is being reported by a nats streaming pod itself, not from a nats streaming client. In the source code I see nats streaming sets the reconnection buffer to -1 (no buffering) so my question is, if this error occurs will nats streaming retry the publication (to nats) or will the message be lost?

[kozlovic]

The streaming server indeed also set this option to avoid issues I described above (but in this case duplicate on deliveries). No, messages are not lost since messages will not be acknowledged by the applications (they did not receive the messages), messages will be redelivered.