Replies: 7 comments 37 replies
-
|
Oh my! Great work, and I think you should also check out the discussion in #754 and #755 — and I'd like to loop in @Coreforge, @RSC-Games, @pauld-gh, and @lts-rad who will also be very interested... I will definitely try to validate the patches on my end too, once I'm back in the studio, this is exciting to see! |
Beta Was this translation helpful? Give feedback.
-
|
HEY WE WERE SO CLOSE So it also looks like from the results above that we were on the right track with the If For AMD testing I forked the RPi Linux kernel and have a recently-rebased version of the older Coreforge patches available at https://github.com/RSC-Games/linux.git... I might actually just merge a PR into Jeff's repo instead so there's a centralized location for all of the patches. |
Beta Was this translation helpful? Give feedback.
-
|
That's neat it works with that little change. Setting the caching to |
Beta Was this translation helpful? Give feedback.
-
|
@yanghaku have you published a PR, a fork of the open-gpu-kernel-modules repo, or the patch set for it anywhere? I'd love to test your patch on the 3080 Ti I have here on a Pi. Will test one or two of my AMD GPUs again soon, too. |
Beta Was this translation helpful? Give feedback.
-
|
Currently, I don’t have a PR or a patch set for submission yet, as the modifications to the open-gpu-kernel-modules are still in a draft state. The changes are quite extensive and complex (affecting about 50 files), and many of the debug additions (such as NV_PRINTF) haven’t been cleaned up. The current version also lacks generality—for instance, I’ve directly changed some cached mappings to uncached, which can only run on the arm64 platform, rather than all. The proper approach should involve checking for DMA coherence first. Additionally, I haven’t verified whether these changes are actually necessary. So far, I’ve only tested the modifications on Ampere architecture (3090). Other architectures (such as Turing, which I plan to test on a 2080 Ti) haven’t been validated yet. I do have other ongoing tasks, but I’ll try to allocate time to finalize the patch set as soon as possible. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for all your work so far, and there's no rush of course. Have you by chance communicated at all with @aritger from Nvidia? Over in NVIDIA/open-gpu-kernel-modules#725 Andy was the one who had asked for more details about our earlier testing efforts. It might be possible Nvidia could also provide at least some assistance too! (Though I think they'd care more about the newer 40/50 series and less about 20/30 (even though those are the models that would make the most economical sense on Pi!). |
Beta Was this translation helpful? Give feedback.
-
|
@yanghaku - I've tested your AMDGPU patch with a 7900 XT 20GB card on a Pi 5 and wow, almost everything I've tested works right out of the gate, no other tweaks necessary. See my testing results here: #763 The one thing I've had some trouble with is some Steam games launching, but that's adding on a bunch more complexity on top of the driver stack, so I can't blame it on any driver issues in particular yet. But wow, what a much easier approach, just a few lines changed :D And the 7900 XT 20 GB card is a pretty good performer for the money, though efficiency isn't the best. At least it's readily available under $700. |
Beta Was this translation helpful? Give feedback.
-
The current update is, the firmware which is there in NVIDIA card, is not being able to communicate properly. Somehow it's not initialising. |
Beta Was this translation helpful? Give feedback.
-
In struct device {
// .......
bool dma_coherent:1;
// .......
};Reading this variable can know if it "has io consistency", and I think no kernel config needs to be modified. |
Beta Was this translation helpful? Give feedback.
-
|
@yanghaku this worked well. What was the build process you used along with building the open modules,. Any specific configurations dtb changes ? Even though you're working on pi, that would help here. Here's what I'm getting. I'm using a rk3588 sbc Here's the patch: +++ b/src/nvidia/arch/nvalloc/common/inc/nvpcie.h +// Rockchip Rk3588 definitions
} This solves pcie config space is inaccessible but still stuck with gsp firmware issue . |
Beta Was this translation helpful? Give feedback.
-
|
The RK3588's lack of support for reasonable BAR sizes is really going to bite you here. AMDGPU requires only 256 MB BAR, which has been supported on x86 since probably before I was born- but the RK3588 doesn't appear to support larger than like a (16MB?) BAR size. That combined with the lack of 64 bit DMA is going to make supporting the RK3588 a massive pain, as bounce buffering is required and GPU drivers (particularly AMD) reaaallly don't like to interact with it, at least without extensive modifications. I remember someone asked about the Pi 4's PCIe limitations, and to the best of my knowledge they are these: |
Beta Was this translation helpful? Give feedback.
-
|
@RSC-Games thanks for addressing the pcie bus issues. @mariobalanica solves the BAR issue [ 9.384277] NVRM: Chipset not recognized (vendor ID 0x1d87, device ID 0x3588) I think the firmware issue is due the it says chipset not recognized. The kernel modules do not contain the vendor ID and device ID of my setup . I think it should be added. The CPU and GPU cache snooping , both of them must be disabled, since it won't work here. I have created a patch, hoping it works with some minor changes. hrushirajg23/open-gpu-kernel-modules@dfc470f some dirty NV_PRINTF statements are also included @yanghaku is it going right way ? |
Beta Was this translation helpful? Give feedback.
-
|
So looking at the patch here... it appears to be a patch to the drm code itself, rather than to |
Beta Was this translation helpful? Give feedback.
-
|
I'll try to test a few other cards tomorrow when I'm back at the studio. Didn't even notice it wasn't in the The sad/annoying thing is if I'm reading the tea leaves right, Intel might not continue making inexpensive consumer graphics cards. It would be sad, because we desperately need more low end competition to Nvidia. |
Beta Was this translation helpful? Give feedback.
-
You're not the only one worried. I went down a small rabbit hole doing research on this hoping intel wouldn't kill the cards. I think ARC is done for too from what I've seen which is really sad bc the cards are amazing. But oh well... Honestly I feel like Intel could've gotten away with 275-300 bucks for the b580 rather than 250, but that's irrelevant. |
Beta Was this translation helpful? Give feedback.
-
|
I've re-tested the B580 and A750 cards (see #764), and they both work great, along with an RX 6500 XT. Still need to do a bit more testing, but the Arc cards did need the small patch for VGA issues, though not the timing patch. And they still showed that weird artifacting, but less I think than last time I tested. |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone tested these changes on an LTS kernel? Home Assistant uses 6.12 and it'd be great to use this tiny patch instead. I wonder though, how does this actually solve the alignment problem, as opposed to just the coherency problem? |
Beta Was this translation helpful? Give feedback.
-
|
Indeed! I've tested this patch in Home Assistant and it works like a charm. I've submitted a PR to upstream those changes, fingers crossed. home-assistant/operating-system#4335 |
Beta Was this translation helpful? Give feedback.
-
|
To my current knowledge we have Intel somewhat working and nouveau is in the works (awaiting the nvidia-open fixes) so amd is the current best bet. However I'm thinking I'll join efforts with Jeff to get these changes into mainline since they'll likely benefit way more boards than just the Pi 5 |
Beta Was this translation helpful? Give feedback.
-
|
Yes if we can get them in mainline Linux that would be incredible. Or even the RPI fork would give us an out of the box experience. I have an a770 to test with but so far it seemed to glitchy for real use on any patch set I've seen so far. Of course cuda is the golden standard for LLMs so if that would someday work out of the box it'd be incredible but that's a long way out. |
Beta Was this translation helpful? Give feedback.
-
|
Noob question @RSC-Games
Since this memory is "non cacheable" doesn't this cause some performance loss? |
Beta Was this translation helpful? Give feedback.
-
|
Hey- noob questions are really good for learning! Don't be afraid to ask them lol- we're all encouraging people to learn and help out.
You are correct. Performance will be lower than on an equivalent system with the device memory mapped as cacheable. However, since the PCIe RC doesn't snoop any AXI traffic, this means writes to memory that land in the ARM cache will never be seen by any other devices. With the Xe kernel drivers enough was happening between each write that lines were being evicted to memory and DMA'd to the card, so we'd see intermittent success, but AMD and Nvidia didn't have those weird race conditions and just failed to initialize. Coreforge's original patchset mapped the PCIe ranges as MMIO (I don't remember the official name for it), which per ARM spec doesn't allow unaligned read/writes, and isn't cached. To fix that, he had to fix up unaligned load/stores with a kernel mode trap so we could break it into multiple aligned read/writes. However, this came with a lot of overhead, especially for games/apps with frequent unaligned memory operations. Then yanghaku came in right about when we all realized that DMA coherence was the main issue at play, and released a 15 line patchset resolving the vast majority of memory issues we know of in the TTM memory allocator- rendering most of the driver-specific patches completely unnecessary. So, between having the memory mapped as |
Beta Was this translation helpful? Give feedback.
-
|
Also of interest to anyone following these topics, @mariobalanica published a set of arm-pcie-gpu-patches specifically geared towards the RK3588 (tested on a Radxa Rock 5B board), which at least has an RX580 working — it looks like there are some BAR adjustments needed on that platform, but one thing that was demonstrated was early boot screens working with UEFI support via AMD's aarch64 GOP driver... See Twitter thread with some more screenshots and details: https://x.com/mariobalanica/status/1977826099104243980 Not sure if that patchset will work on RK3588 with AMD GPUs with more than 4 GB of VRAM though? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello @geerlingguy ,
Thank you so much for your useful and awesome work! For my research work, I was greatly inspired by your open-source AMD GPU driver patch. Through this, I discovered that the root cause is that most ARM SoCs lack I/O coherence, while many PCIe device drivers (such as the AMD GPU driver) are primarily designed for x86 platforms and implicitly assume DMA coherence in certain implementations.
Based on this finding, I have successfully enabled both AMD and NVIDIA GPU support on Raspberry Pi 5B in our tests.
AMD GPU (Sapphire RX 460 OC 4 GB)
My Patch
This is my simpler patch on the AMD GPU driver based on Linux raspberrypi 6.15.0, there are just two changes:
kmapcalls with unifiedvmaplogic in memory managementpgprot_dmacoherent()for ARM64 architecturesExperimental setup
AMD GPU (Sapphire RX 460 OC 4 GB) on a Raspberry Pi 5 with the 52Pi PCIe x1 Slot HAT :
Then I connected PCIe TO 2-CH PCIe HAT and PCIe to USB3.2 Gen1 Board to support more usb devices:
Results
dmesglog about amdgpu:lspci:vulkaninfo,glxinfoandglxgearswith X11 display:glxgearsDisplay on screen:NVIDIA GPU (GeForce RTX 3090)
My Patch
I modified Nvidia Open-Source driver for Linux raspberrypi 6.15.0, The patch is a bit complicated, involving multiple files.
Experimental setup
To use a more stable high-power PCIe slot for the power supply, we used an OCuLink GPU Dock, the product is ADT-Link F9G-BK7-F9934-F4C, connected to the Raspberry Pi via the Raspberry Pi M.2 HAT+
Results
dmesglog about nvidia gpu:nvidia-smiandlspci:vulkaninfo,glxinfoandglxgearswith X11 display:glxgearsDisplay on screen:Some CUDA Samples:
Run a LLM (use mlc-chat):
This
nvidia-smiresult when running the LLM above:nvidia-smi and lspci on version 570.133.07 (an old picture):
Beta Was this translation helpful? Give feedback.
All reactions