Got cutlass error: Error Internal at: 171

[dhunganaprakas]

Hello, I am currently trying to reproduce your training instance in a Windows PC. The development setup is mentioned as below:

OS: Windows Pro
CUDA Toolkit: 11.0
Graphics Card: Quadro P620 (Pascal Architecture - sm_61)
CUDNN Version: 8.2.0
Python: 3.8.3
Torch: 1.7.1+cu110

**The paths for Windows are adjusted in setup.py script for all cuda extensions. To address the graphics card architecture I replaced the using SmArch = cutlass::arch::Sm75 with using SmArch = cutlass::arch::Sm61 in source code file niti\pytorch\cutlass-extension\cutlassconv_kernel.cu (7 occurences)

I can see the installed packages in this location after successful make: C:\Users*Username*\AppData\Local\Programs\Python\Python38\Lib\site-packages

When running make architecture is confirmed by the following line at the end of the compile command. (-gencode=arch=compute_61, code=sm_61)

When training I came across "Got cutlass error: Error Internal at: 171" error.

Can you please check the issue and propose an appropriate solution.

[wangmaolin]

Pascal architecture is too old and doesn't have tensor cores.

[dhunganaprakas]

Is there python function (instead of CUDA extension functions) to avoid using CUDA and Tensors at all? If there is, can you say if the training time is increased significantly?