CM5 Super6C working

[pcnate]

I have swapped 2 of the 6 CM4s in my Super 6C (see #425) to run on CM5. Let me know what commands are typically used to provide details about the health of the OS and compatibility of the hardware and I will post it here.

Important note on the physical mounting. The CM5 official heatsink is not compatible with the Super 6C. There are threaded studs on the Super 6C so you screw down through the CM4 heatsink into the threaded studs. On the CM5 heatsink, there are threaded holes on the bottom side of the heatsink. I ended up using the CM4 heatsink with some extra thermal pads to cover the slightly different layout of hardware.

My setup used some NVMe drives that had the OS installed on them. I simply swapped the CM5 and let the same software bootup. I figured it was worth a shot as I've seen this work before between Pi Zeros and 3 or 4s (can't remember)

[dvdmuckle]

Curious about something, what are you using for power? I have a similar setup with three CM4s and two CM5s (with a third ready to install) that are seeing some high IOWait booting off of the same Inland 256GB SSDs; 15% on the CM5s, and around 2-3% on the CM4s. I'm thinking it may be to do with power, and that the CM5s aren't getting enough power for both the SSDs (running at gen3 speeds) and the CM5s themselves. I'm going to test this with an old PSU soon, as the included adapter only handles around 90W, but I'd like to hear your experience with this as well.

[dvdmuckle]

I just got in an adapter for my PSU and have said PSU installed in my cluster. I'm going to see if this helps the IOWait I've been seeing. There's a good chance it doesn't, from early looking...

[dvdmuckle]

Unsure if you've had similar issues with your SSDs, sounds like you haven't. But I am now using an ATX PSU and the following in my cmdline.txt for my CM5 nodes, and those SSDs and nodes seem a lot more stable now:

nvme_core.default_ps_max_latency_us=0 pcie_port_pm=off pcie_aspm=off

Unsure if all of those things are necessary or not, but it works so I'm a bit hesitant to touch it further.

[pcnate]

what were you using to measure your iowait issues?

[dvdmuckle]

Prometheus + Grafana on a K3s cluster. But I was also using Netdata, which is much easier to install and set up. You could also use top but I like Grafana and Netdata for the historical graph.

[dvdmuckle]

For what it's worth as well, before setting those cmdline args, I was seeing a lot of Longhorn volume rebuilds, which would impact Prometheus and cause it to be non-functional unless I deleted the pod.

[JPL4494]

@pcnate @dvdmuckle What OS are you using for the pis? I've had nothing but trouble with Raspbian, the NVMEs aren't showing up, and the USB ports don't work. It's sounding like those aren't compatible on the Super6c since some of the wiring is different compared on the CM5 compared to the CM4. I couldn't get Ubuntu 25.10 to boot and stay running, it kept freezing on me

[pcnate]

An important note, I did not install the OS onto the CM5. I did so on the CM4 and the later swapped the CM4 out for the CM5 and it has not had issues. I'll have to look later as it appears my cluster is offline, forgot to check its status after a power outage the other day. I think it is raspbian.

[JPL4494]

Got it, so you installed it on the NVME or SD card then? I installed it on the eMMC, so that could be the weird difference that's causing me issues.

FWIW, deskpi is claiming that the USB and NVME don't work with the CM5 at all. If you got it working, I'll mirror your setup so I can stop pulling out my hair, haha

[pcnate]

I'm running the Lite version. I started using the sdcard then moved the data onto an NVMe. Then I swapped 2 of the 6 CM4s for CM5s.

[JPL4494]

Got it, thanks! Last question, I swear - are you using a specific brand/model for the drives? I (maybe stupidly) thought as long as it was the correct type of M.2, it would work. But, I'm seeing pi's are a little picky with compatible drives