Fix biometrics on Pod 5: cbor2 C extension skips records + empty placeholder records break read loop

[seanpasino]

Problem

Biometrics produce no vitals data on Pod 5 due to two separate bugs in load_raw_files.py and stream.py.

---

Bug 1: cbor2.load(f) skips nearly every record

When the _cbor2 C extension is installed (the default when cbor2 is pip-installed with a compiler available), cbor2.load(f) reads files in internal 4096-byte chunks. Each call advances f.tell() by 4096 bytes regardless of how large the actual CBOR record was.

RAW file records are typically 17–5000 bytes each. This means only one record per 4096-byte block is decoded — all others are silently skipped. On Pod 5, piezo-dual records are ~2700 bytes, so the stream sees almost no data. On Pod 3/4 with larger records (~5000 bytes across 4 sensors), the effect is smaller but accuracy is still degraded.

Before (both files):

row = cbor2.load(f)          # f.tell() jumps by 4096 regardless of record size
decoded_data = cbor2.loads(row['data'])

After:

data_bytes = _read_raw_record(f)   # f.tell() advances by exact record size
decoded_data = cbor2.loads(data_bytes)

_read_raw_record(f) manually parses the outer {seq: uint, data: bytes} CBOR wrapper byte-by-byte using f.read(), keeping the file position accurate.

---

Bug 2: Empty placeholder records raise EOFError mid-file, stopping the loop early

Pod 5 firmware writes placeholder records with data = b'' (CBOR byte 0x40 = empty byte string) as sequence number markers between real data records. Calling cbor2.loads(b'') raises CBORDecodeEOF, which is a subclass of EOFError.

Both files catch EOFError to detect end-of-file and break the read loop. So any placeholder record mid-file terminates the entire read — even with megabytes of valid data remaining.

Fix: _read_raw_record() returns None for empty data; callers continue.

---

Additional fix in stream.py

The original except Exception handler called break, leaving f.tell() at an unknown position. Changed to seek(last_pos) before breaking so the next poll cycle starts from the last known good position.

---

Files changed

• biometrics/load_raw_files.py — added import struct, added _read_raw_record(), replaced cbor2.load() call
• biometrics/stream/stream.py — added import struct, added _read_raw_record(), rewrote follow_latest_file() read loop

---

Tested on

• Device: Eight Sleep Pod 5
• cbor2: 5.6.5 with _cbor2 C extension active
• Python: 3.10

Result: 846 vitals rows (heart rate, HRV, breathing rate) and 400 movement rows recorded in a single night. Stream runs stably with no OOM kills or error floods.

---

Backward compatibility

The fix is fully compatible with Pod 3/4. The outer {seq, data} CBOR format is identical across all pod versions. Pod 3/4 records with left2/right2 fields are unaffected — inner record decoding is unchanged. Pod 3/4 may also see improved accuracy as more records are now correctly decoded.

[vercel]

Someone is attempting to deploy a commit to the david's projects Team on Vercel.

A member of the Team first needs to authorize it.