Track peak row memory usage in Thor activities

[Copilot]

Description

Thor activities that buffer rows using thorRowMemoryFootprint() now track peak memory usage to enable memory capacity planning for job execution.

Changes

Buffer classes enhanced with peak tracking:

• CSmartRowBuffer, CSmartRowInMemoryBuffer - Added peakInsz member, updated on each putRow()
• CCompressedSpillingRowStream - Added atomic peakInMemRowsMemoryUsage, thread-safe updates via compare-exchange
• CSharedFullSpillingWriteAhead - Added peakRowsMemUsage member
• CParallelFunnel - Added peakTotSize member, tracked in push() and pushMulti()

Interfaces extended:

• ISmartRowBuffer, ISharedRowStreamReader, IStartableEngineRowStream - Added getPeakRowMemory() method
• Stub implementations return 0 for classes not using thorRowMemoryFootprint()

Statistics publishing:

• CSlaveActivity::gatherActiveStats() - Publishes StSizePeakRowMemory from input look-ahead streams
• FunnelSlaveActivity - Publishes peak from parallel funnel buffer
• NSplitterSlaveActivity - Publishes peak from shared row stream

Peak values are captured wherever memory footprint is calculated, requiring no changes to existing buffer logic beyond the tracking comparisons.

Type of change:

• This change is a new feature (non-breaking change which adds functionality).

Checklist:

• My code follows the code style of this project.
  • My code does not create any new warnings from compiler, build system, or lint.
• The commit message is properly formatted and free of typos.
  • The commit message title makes sense in a changelog, by itself.
  • The commit is signed.
• My change requires a change to the documentation.
• I have read the CONTRIBUTORS document.
• The change has been fully tested:
  • I have added tests to cover my changes.
  • All new and existing tests passed.
• I have given due consideration to all of the following potential concerns:
  • Scalability
  • Performance
  • Security
  • Thread-safety
  • Cloud-compatibility
  • Premature optimization
  • Existing deployed queries will not be broken
  • This change fixes the problem, not just the symptom
  • The target branch of this pull request is appropriate for such a change.
• There are no similar instances of the same problem that should be addressed

Testing:

Implementation adds minimal tracking overhead - single comparison per row buffered. Thread-safe atomic updates used for CCompressedSpillingRowStream. Existing StSizePeakRowMemory statistic merges via max as required.

Original prompt

When a thor activity buffers rows in memory it uses the thorRowMemoryFootprint to calculate how much memory they are consuming. We need to be able to understand the peak memory usage of these activites in order to work out how a thor job would run if the memory was reduced.

Implement the following, creating one or more commits for each of the steps:

• For all classes that call thorRowMemoryFootprint() add a member variable to track the peak memory usage. They are likely to already track the current memory usage, so the code should be relatively small. Add a member function to access the value.
• For all activities that use those classes, publish a StSizePeakRowMemory statistic for the activity - which corresponds to the high-water-mark. Merging that statisic will already ensure it is the maximum, so no special code is needed.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.