New jcstress concurrency stress testing module

[the-thing]

Introduce jcstress module for concurrency stress testing.

Changes

• new quickfixj-stress-test for manual concurrency stress testing
• some minor refactoring to quickfix.JdbcStore to allow testing
• minor formatting fixes

Notes

• the tests are built as part of package stage, but are never run automatically
• this PR doesn't fix anything except formatting and renamed Maven property value

I wrote two tests intially. One for quickfix.JdbcStore and one for quickfix.MemoryStore as a starter.

Each test encapsulates only a MemoryStore implementation, but both tests assume that sequence increment is guarded by their respective reentrant lock (which I'm not entirely sure about that this happens everywhere in the code).

Stress test clearly shows that there are concurrency issues in quickfix.JdbcStore implementation - #357. See commens in JdbcStoreStressTest.java. This errors are rare - < 0.01% under specific circumstances.

I think there might be more issues related to reading sequences by different threads e.g. "QFJ Message Processor" thread reading stale sender sequences modified by application threads, but it is hard to confirm.

I hope that I got the mocks right for quickfix.JdbcStore test. Let me know if you see any value in this.

[chrjohn]

Hi @the-thing , thanks for this, looks very useful. I also recalled that issue #357 when reading your description.
Does it make sense to have some kind of scheduled github action for the stress test?

[the-thing]

I'm not sure about the CI integration.

The issues I can see are

• these stress tests take a long time to run
• there is no standard way to capture the results (at least I'm not aware of), but the reports in are generated in the console and html files
• the action would have to wait for main pipeline to finish and download jcstress module files and run them (I think this is relatively easy to do)

I don't think I have seen jcstress running as part of CI pipeline before. Normally when one suspects a concurrency bug. You write jcstress test, fix it and rerun. Having it as part of the project makes things easier with manual re-runs and setup.

However, I have seen benchmarks being run automatically for continuous performance monitoring.

It depends how you are planning on using it. I imagine there is another job that runs after the main CI build that always succeeds and if anyone is interested can view the results on demand. Is that what you are after?