Storage interrupts sampling

[schmettow]

Saving to SD takes much time, leading to missed sensor samples.

Quick fix: Increase the saving interval.

Two possible strategies:

1. Make saving faster by keeping everything mounted (currently, it runs the full cycle)
2. Using the second core on RP2040 to split the tasks on two cores. Requires multi-threaded computing.

[schmettow]

Attempts to improve the writing speed have largely failed. Speed tests show that writing is limited to ~ 14 kb/s, which is painstakingly slow and duie to the SPI bus.
Increasing the saving interval doesn't help much, because it just makes the writing longer and one quickly risks memory allocation errors.
Putting storage on the second core of the RP2040 would be the natural solution, but that is the one thing CircuitPython cannot do (MicroPython can).
As a quick alleviation I made a new storage device BSU (Burst Saves over USB), which sends CSV data up the USB line. A desktop program can catch and process the stream. USB seems to be faster, but further tests are required.
For advanced setups with high data throughput, we can also look at Raspberry Pi Zero, which has a much faster SD card interface (SDIO) and is more power overall.

[Karel-Kroeze]

Wouldn't *decreasing *the saving interval be a better option? If the data to write is small enough and the write operation fast enough to fit in the interval between measurements, that would cleanly solve the problem. That would probably require creating a file for each sensor, but that shouldn't be a major issue?

…

On Wed, Jan 4, 2023 at 9:37 PM schmettow ***@***.***> wrote: Attempts to improve the writing speed have largely failed. Speed tests show that writing is limited to ~ 14 kb/s, which is painstakingly slow and duie to the SPI bus. Increasing the saving interval doesn't help much, because it just makes the writing longer and one quickly risks memory allocation errors. Putting storage on the second core of the RP2040 would be the natural solution, but that is the one thing CircuitPython cannot do (MicroPython can). As a quick alleviation I made a new storage device BSU (Burst Saves over USB), which sends CSV data up the USB line. A desktop program can catch and process the stream. USB seems to be faster, but further tests are required. For advanced setups with high data throughput, we can also look at Raspberry Pi Zero, which has a much faster SD card interface (SDIO) and is more power overall. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABUD37D2Z2RZPL2IXYMALH3WQXNO5ANCNFSM6AAAAAATFPRUFI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[schmettow]

I tested continuous writing by setting save_interval = 0 and trigger with every update. In result the sampling rate went down to 20 sps. The baseline with no storage is 250. That's probably because the overhead of excessive open/close kicks in. Don't forget that all file system management is done in Python.

The new BSU storage interface sends data upstream. Not tested yet, but indirectly we know that printing is not delaying sampling by much. You can try out the buzzwire app. It is like YLab Zero, but adding a send mode (long press during Stop).

[Karel-Kroeze]

Could you not keep the file open and keep a handle around? Just thinking aloud here :). I'm not sure what the expensive part of the operation is here, but if opening/closing the file takes that long - we should probably minimize the amount of times we're doing it?

[schmettow]

First, I also thought it to be the overhead. But as I said, writing itself is limited to 14kB/s. We are saving numbers as strings, that means a single sensor reading is already more than 50 B with 16 bit encoding. If we sample at 250Hz, saving one second of data almost takes a second.

That means we can massively reduce the writing by using a byte format, reducing every row to two 32bit floats for time and value and one 8bit integer for the sensor number. That should reduce the whole throughput by factor 10. In order to map sensor number to label, we can simply produce csv meta data alongside.

If we figure out how to write data lines as bytearrays, this mechanism should be easy to derive from the SDCard class. At a later point, we can splice Ydata and inheriting classes into separate Handler (SDCard, BSU) and Writer (CSV, Byte) classes and derive all possible combinations by multiple inheritance.

Anyone?

[Karel-Kroeze]

I hadn't realized the write speed itself was actually the bottleneck, but in that case storing binary data makes sense. Some initial thoughts (in more or less random order):

• as bytes/floats/ints/whatever will have a fixed length, we shouldn't need data structures (e.g. bytearrays). Simply parse in and out with the same format (byte packing).
• Byte packing/unpacking is straightforward in python, and shouldn't be too hard to replicate in R.

We may want to consider storing raw data from the sensors, and doing any processing on the 'host' device.

• You mention 32bit floats, but if I recall correctly that's a higher precision than the sensors themselves provide. Writing raw sensor data would minimize the required bitrate and be lossless (floats have rounding errors).
• For some sensors, it may be possible to discard bits that aren't interesting (i.e. outside of a reasonable values range. It's also likely the dev boards with built in amplifiers already do this themselves).
• We'd need to reconstitute the 'real' values from the raw data on the 'host' device, which implies either coming up with a way to structurally describe how to reconstitute data (e.g. bits per value, baselines, scaling, etc.) and storing that as metadata, or maintaining a library of host device code to do so for specific languages (R, Python) and sensors.

It's worth keeping in mind that the nitty gritty of data formats probably goes well beyond the average students' understanding. I didn't find it too hard to figure out when I was working out communications with simple device interfaces (I2C, SPI) from scratch, and in theory it could be worthwhile for their understanding (it certainly was for me), but it's well down the rabbit hole.

[Karel-Kroeze]

As for the specifics of how to implement that, composition (which I think for python is pretty much the same as multiple inheritance) seems like a good approach - assuming we want to keep different options for storing/handlers/etc. open.