Finilize State Sync

[volovyks]

Description (updated)

Design: https://github.com/sig-net/mpc/blob/develop/doc/mpc_node_specification.md

Success metrics:

• Same number of Total on all nodes Total should not diverge without a good reason
• Sum of mine triples and pre-signatures on all nodes is equal to 'total' Ps and Ts Same, can be different, but this is a good metric to understand how fragmented the storage is.
• Sync is triggered according to design

Related comment: #649 (comment)

[volovyks]

@jakmeier do we want to remove regular sync requests? Currently, they are scheduled every 24 hours.
Not having regular sync requests is part of the design, but I'm not completely convinced by this idea. State can be lost at any moment, not only during reboots. Nodes can stay online for months without updates or restarts.
Do we want to make sure we are not “hiding” issues by relying on regular sync calls?

We can add more triggers to set the node as "NeedSync".

[jakmeier]

The original reason against having regular syncs is to avoid race conditions.

For example, when a node sends a list of local Ps, a new P owned by the sender could be produced before the message arrive on the other node. That other node has the P but concludes that it should delete it's local P because the owner of it didn't announce having it.

I think there are many such cases that we don't handle properly at the moment.

By syncing at boot, we can ensure sync only happens while at least one side is not participating in the MPC protocols, yet.

State can be lost at any moment, not only during reboots.

Hm, so how would state be lost if not during a reboot?

[volovyks]

@jakmeier I see your point, ok

Redis is accessible to the server owner directly; they can execute any command, including the removal of Ts and Ps.

Owerall I do not mind removing scheduled syncs right after I merge my current PR.

[jakmeier]

Ok, yeah let's remove it then. I don't think we can handle the case of a general corruption of the DB anyway.