Logs are getting flooded by this

[shortthefomo]

The logging of this line can be excessive at times and causing issues with HPCORE and then causes further issues down on to the reputation contract as discovered by @gadget78

hpcore/src/comm/comm_session.cpp

Line 108 in 7c97aaa

LOG_WARNING << "Failed to enqueue comm msg.";

we are unable to edit this value via some .cfg and it has been hardcoded to INFO level
https://github.com/EvernodeXRPL/sashimono/blob/be0039d2b3953c490895f8c3aae04c1d53617526/src/conf.hpp#L63

expected out come a .cfg is created where these values can be overridden by the operator or the default log level is reduced to ERROR.

[shortthefomo]

have since found that there is a .cfg defined here https://github.com/EvernodeXRPL/sashimono/blob/be0039d2b3953c490895f8c3aae04c1d53617526/dependencies/hp.cfg

we need to dial that log level down to err

but still unsure how to even build and deploy this stuff locally on our production instance to validate this.

[shortthefomo]

After altering the cfg of /etc/sashimono/contract_template/cfg/hp.cfg change the log level to err the scores on my two machines have climbed. However that does not solve the core issue, that after the logging there needs to be a small delay.

So altering this line

hpcore/src/comm/comm_session.cpp

Line 106 in 7c97aaa

if (!enqueued)

to

if (!enqueued)
{
   LOG_WARNING << "Failed to enqueue comm msg.";
   // Wait a small delay before allowing it to try again, else we end up flooding the logs
   util::sleep(5);
}

will solve the log flooding issue, I can't fork and push to this repo so someone with access will have to implement this.

I think there is still further work to be done on this issue, as the queue is overflowing and I assume that a websocket is servicing that queue. The websocket in this case should probably be closed and it should reattempt a new connection?

[gadget78]

im going to attempt to build this, with a small alteration to your suggestion,
where i log the message too, as this may help debug the core issue of why this cascade event happens ..

if (!enqueued)
{
    // Convert the std::vector<char> to a std::string for logging.
    std::string msg_str(msg.begin(), msg.end());
    LOG_WARNING << "Failed to enqueue comm msg: " << msg_str;
    util::sleep(5);
}

better handling of the que of course would also help,
maybe some kind of exponential back-off/delay system, as apposed to a fixed amount could help ?
i will try build, and test this first,
then see what these message are, if they just standard message as such, and the root issue is further up the line,
then this will only be a sticky plaster at best. but we have to start somewhere ...

[rippleitinnz]

Fundamental issues of the Reputation contract being out of sync with consensus currently along with the Stage consensus bug issue, https://github.com/EvernodeXRPL/hpcore/issues/401 exacerbates the enqueued message failures along with the Duplicate log messages.

In addition, this setting is probably too limiting given the Peer messaging requirements of consensus and any contract HP is interacting with.

constexpr uint16_t MAX_IN_MSG_QUEUE_SIZE = 255; found in https://github.com/EvernodeXRPL/hpcore/blob/7c97aaa18f4a9c8890aa2e965eba4747412077d2/src/comm/comm_session.cpp#L13