neighbors_convert.so has a dependency on libtorch_python.so

[Luthaf]

This means it can not be loaded as a TorchScript extension in environments where libtorch_python.so is not available, such as when trying to run MD with LAMMPS.

Steps to reproduce:

$ pip install git+https://github.com/spozdn/pet@neighbors_convert_cpp

# on (aarch64) linux:
$ ldd virtualenv/lib/python3.12/site-packages/pet/neighbors_convert.so
	linux-vdso.so.1 (0x0000e0f845f9e000)
	libc10.so => not found
	libtorch_cpu.so => not found
	libtorch_python.so => not found
	libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000e0f845c00000)
	libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000e0f845ed0000)
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000e0f845a40000)
	/lib/ld-linux-aarch64.so.1 (0x0000e0f845f61000)
	libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000e0f845990000)

# on macOS:
$ otool -L virtualenv/lib/python3.11/site-packages/pet/neighbors_convert.so
virtualenv/lib/python3.11/site-packages/pet/neighbors_convert.so:
	@rpath/libc10.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libtorch.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libtorch_cpu.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libtorch_python.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)

This does not come from PYBIND11_MODULE, since it is commented out in the code, so my best next guess is that this comes from the build process/torch's CppExtension, which might be adding linker flags when they are not needed.

pet/setup.py

Lines 22 to 25 in 8061f88

	CppExtension(
	name="pet.neighbors_convert", # Ensure this matches the package structure
	sources=["src/neighbors_convert.cpp"],
	),

Ping @DavideTisi and @abmazitov who had this issue when trying to use PET. A temporary workaround on linux is to remove the missing dependency with patchelf:

patchelf --remove-needed libtorch_python.so path/to/neighbors_convert.so

[spozdn]

@Luthaf what do you think of simply putting "os.system('patchelf --remove-needed libtorch_python.so path/to/neighbors_convert.so')" to setup.py?

[Luthaf]

This feels extremely hacky to me, but it is your project 😃. Also this will not work on macOS.

When looking a bit deeper into this with @abmazitov, the issue does come from CppExtension, and in particular this line: https://github.com/pytorch/pytorch/blob/207564bab1c4fe42750931765734ee604032fb69/torch/utils/cpp_extension.py#L965

A less hacky solution IMO would be to duplicate this class in setup.py, removing the offending line.

[spozdn]

yeah, looks better. Or, one can directly call setuptools.Extension.