Improved memory handling by exploiting lazy memory allocation

[Toreil]

The maximum sequence duration is limited by the available system memory, with a sequence taking around 10GB of memory per minute, limiting sequenced duration to a maximum of around 20 minutes (systems typically max out at 256GB of RAM). In this PR the way that the sequence is stored in memory is changed so that memory consumption on typical sequences is substantially reduced (around 95% for the TSE example script) by exploiting Python/Numpy's lazy memory allocation.

Currently the sequence is stored in memory as one array for a 10 minute sequence the size of the array will be around 100 GB. When the array is created Python checks if that array could fit in memory and reserves a memory block of that size but it doesn't occupy all of the 100 GB because Numpy arrays are lazily allocated meaning that the memory is only allocated once it's accessed. Accessing a sub-section of the array only allocates that sub-section of the array, it doesn't allocate the rest of the array but once a sub-section of the array has been accessed it will remain allocated until the entire array is deleted from memory.

The outputs of most MR sequences are sparsely populated; the RF channel will typically only be transmitting less than 1% of the time, and if sequences have a long repetition time the gradient outputs are also mostly zero (shim offsets are handled separately). The sequence provider only accesses the sequence array in places where the output is non-zero (and therefore the memory is only allocated in a few places), which means that while 100 GB is reserved for the sequence array the actually occupied memory is a few GB, however as the sequence is being played out by the console the entire array is being read (accessed) which means that by the time the sequence has finished playing out the entire array has been accessed and the 100 GB array is now fully allocated in memory, despite initially only occupying a few GB.

In this PR the sequence array is segmented in to a list of 'notify sized' arrays in the sequence provider, the tx_device now goes through the list and plays out these array subsections and deleting from the list as the sequence plays out so that once the memory has been allocated for that 'notify sized' array and it's been passed to the card the array is released from memory. This means that memory consumption peaks immediately after the sequence has been unrolled and is essentially determined by how sparsely the output of the Tx Card is in a sequence. For sequences where the outputs are always on this update doesn't improve memory performance, for sequences with a very long TR this can reduce memory consumption immensely.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/console/interfaces
acquisition_data.py	174	93	47%	95–98, 109–110, 128–129, 144–147, 152–287, 298, 302, 320
acquisition_parameter.py	97	16	84%	84–88, 110, 128, 130, 138, 175–182, 191–193
device_configuration.py	72	10	86%	34–36, 38, 46–47, 55–56, 83–85, 87
dimensions.py	34	9	74%	30, 35, 41–44, 49, 55, 60, 65
rx_data.py	79	14	82%	66–70, 78, 90, 94, 96, 104, 115, 123, 126, 129
src/console/pulseq_interpreter
sequence_provider.py	363	233	36%	97, 99, 101, 103–105, 127, 174, 176, 179, 184, 187–189, 198–200, 233–236, 264–267, 281–282, 313–398, 502, 508, 513–517, 541–784, 812–855
src/console/spcm_control
abstract_device.py	79	79	0%	2–149
acquisition_control.py	142	142	0%	3–343
rx_device.py	205	205	0%	2–484
tx_device.py	190	190	0%	2–491
src/console/spcm_control/spcm
pyspcm.py	143	143	0%	3–276
tools.py	54	54	0%	3–128
src/console/utilities
json_encoder.py	8	3	62%	22–24
load_configuration.py	20	6	70%	18–20, 28–30
plot.py	17	17	0%	2–26
src/console/utilities/sequences/calibration
fid_tx_adjust.py	18	1	94%	56
se_tx_adjust.py	25	2	92%	56–57
src/console/utilities/sequences/spectrometry
fid.py	15	15	0%	2–70
se_spectrum.py	25	2	92%	48, 72
src/console/utilities/sequences/tse
tse_3d.py	241	75	69%	108–109, 111–112, 114–115, 122–123, 125–127, 129–132, 143–146, 151–154, 161–172, 189, 211–218, 224, 255, 318–335, 342–343, 410, 413–425, 533–544
TOTAL	5246	1309	75%

Tests	Skipped	Failures	Errors	Time
171	0 💤	3 ❌	0 🔥	6.952s ⏱️

[Toreil]

Inherent to the way that this method works is that the unrolled sequence is no longer in memory after the sequence has been run, this clashes with how averaging is done in console, which is on a console level, rather than a sequence level. Since Pulseq supports flags for averages and it's relatively trivial to add to the inbuilt sequences I propose removing the console level averaging in favour of averaging defined in the .seq file, particularly since the motivation behind console level averaging was the inability to handle long sequences due to memory consumption which is (for the most part) addressed in this PR.