[SPARK-54363][TESTS] Access to buffer in SparkShellSuite should be synchronized between threads

[vrozov]

What changes were proposed in this pull request?

Synchronize access to buffer in SparkShellSuite using existing lock.

Why are the changes needed?

buffer is updated on ProcessOutputCapturer threads and when it is access during exception handling, access to the buffer needs to be locked to avoid using partially updated buffer.

Does this PR introduce any user-facing change?

No

How was this patch tested?

running test

Was this patch authored or co-authored using generative AI tooling?

No

[sarutak]

@vrozov How about just using a concurrent collection like LinkedBlockingQueue for buffer?

[vrozov]

@vrozov How about just using a concurrent collection like LinkedBlockingQueue for buffer?

@sarutak Thank you for suggestions. The PR uses existing lock that will be still required even if buffer is changed from ArrayBuffer to LinkedBlockingQueue to synchronize between ProcessOutputCapturer for stdout and stderr. Additionally, queue usage pattern (take one element at a time from head or tail) does not apply well here as it outputs the entire buffer (collection).