Possible performance degradation under mutex contention

[voximity]

We have been considering the possibility of mutex contention causing performance degradation in snarkOS and snarkVM.

The two ecosystems make use of parking_lot for its Mutexes and RwLocks. It is known that parking_lot suffers from performance degradations when under heavy contention. We could confirm this when running the benchmark on a machine that runs a validator on the Canarynet.

Steps to Reproduce

N/A. We are looking into analyzing the prevalence of contention, as well as experimenting with replacing the usage of parking_lot with std::sync::Mutex.

Expected Behavior

N/A.

Your Environment

N/A. We tested the benchmark above in Ubuntu on WSL and Ubuntu on bare metal.

[ljedrz]

The fact that these days std locks are faster is true indeed; I'm assuming this (performance degradation) hasn't been observed yet? FWIW, no lock contention has popped up in perf profiles I've run before, but perhaps I wasn't testing in extreme enough conditions. Also, I tested std locks against parking_lot ones in a stress test for an unrelated network app in the past, but wasn't able to detect a difference in performance (though perhaps that app had fewer lock-related operations involved).

[zosorock]

how can we monitor this @ljedrz ?

[ljedrz]

with periodic (but not necessarily long) perf runs; high lock contention should be apparent there