Discuss how to set up stress testing of the system

[CGodiksen]

An environment could be set up that makes it possible to start a number of edge and cloud nodes. A script should be created to ingest an adjustable amount of data into the nodes. Another script should be created to simulate users using the system. The script should make it possible to adjust the amount of users, how many requests a single user sends every minute, and the distribution of what requests the user makes.

Ideally the environment could be run for some period of time and metrics describing the failure rate, average response time, and more could be presented when the stress test is finished.

Maybe take inspiration from existing stress testing frameworks such as this: https://blog.postman.com/postman-api-performance-testing/

We should also look into how to handle stress testing that ensures that the WAL works and correctly recovers to avoid data loss and continue operations that were in process when the system crashed.

[CGodiksen]

Instead of setting up a custom system for stress testing we could also rely on existing functionality for continuous testing with simulated failure:

Hardware never performs their tasks 100% correct, e.g., storage devices fail writes/reads and corrupts data over time. Likewise the network connection may corrupt packages, degrade to very low amounts of bandwidth and very high latency, or simply fail. Possible solutions for this could be fault-injection or simulation:

• https://apple.github.io/foundationdb/testing.html
• https://blitiri.com.ar/p/libfiu/
• https://dropbox.tech/infrastructure/-testing-our-new-sync-engine
• https://github.com/awslabs/shuttle
• https://github.com/madsim-rs/madsim
• https://github.com/tigerbeetledb/tigerbeetle
• https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/HACKING.md
• https://github.com/tigerbeetledb/tigerbeetle/tree/main/src/vopr_hub
• https://github.com/tokio-rs/loom
• https://github.com/tokio-rs/simulation
• Consider exposing simulation's types in Tokio tokio-rs/tokio#1845
• https://github.com/tokio-rs/turmoil
• https://highassurance.rs/
• https://medium.com/@RisingWave_Engineering/applying-deterministic-simulation-the-risingwave-story-part-2-of-2-520122d371db
• https://medium.com/@RisingWave_Engineering/deterministic-simulation-a-new-era-of-distributed-system-testing-part-1-of-2-69f151593a1
• https://model-checking.github.io/kani/tool-comparison.html
• https://rust-formal-methods.github.io/welcome.html
• https://tokio.rs/blog/2023-01-03-announcing-turmoil
• https://www.inapps.net/update-chaos-tools-and-techniques-for-testing-the-tidb-distributed-newsql-database/
• https://www.youtube.com/watch?v=OJb8A6h9jQQ