fragile SAGE_LIB in setup.py

[videlec]

The assumption about the localtion of SAGE_LIB might be wrong

SAGE_LIB = f"{SAGE_LOCAL}/lib/python{PYTHON_VERSION}/site-packages"

On my system the compiled version of sage does not satisfy the above

$ sage-dev -python -c 'from sage.env import SAGE_LOCAL, SAGE_LIB; print(f"{SAGE_LOCAL}\n{SAGE_LIB}")'
/opt/sage/local
/opt/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages

Part of openjournals/joss-reviews#3996

[fredstro]

@videlec do you have any suggestion for fixing this?

Importing SAGE_LIB from sage.env is fine for just doing sage setup.py install but for a source installation
i.e. sage -python setup.py sdist followed by sage -pip install sdist/* it doesn't work due to build-isolation .
The only way around seems to use the --no-build-isolation flag, i.e.
sage -pip install dist/* --no-build-isolation
but I don't think it is reasonable to ask people to use a special flag, or what do you think?

[videlec]

By providing the SAGE_LIB you are anyway bypassing silently the build isoliation. The only way to support the build isolation would be to set SageMath as a build dependency which is of course not viable.

The --no-build-isolation looks like a better option. Isn't it possible to force this option inside setup.py and or setup.cfg?

You could possibly ask the question on sage-devel as I am not an expert of these Python packaging subtleties.

[videlec]

The discussion at pypa/pip#5033 is relevant.

[videlec]

Also pypa/pip#6718

[fredstro]

I am still looking into this but it appears it is not possible to bypass the build isolation from any configuration file.

It seems to be an explicit intention in the current pip development that package authors should not be able to do this, only end users.
The only option seems to be to either

1. Make an spkg package instead
2. Ask users to supply a '--no-build-isolation' flag
3. Try some other method to locate the library files in the file system (look at all possible relative locations to SAGE_LOCAL)

What I have tried and unfortunately doesn't work is to use something like

cmd = "import sage; from sage.env import SAGE_LIB; print(SAGE_LIB)"
sage = shutil.which('sage')
proc = subprocess.run([sage, '-c', cmd], capture_output=True)
SAGE_LIB = proc.stdout.decode('utf8')

It works from a standard python prompty but somehow the isolated environment that pip creates makes it so that the sage library is not found even from within the sage executable...

I still haven't asked on sage-devel but will try to do that soon. I just wanted to try all options I could think of myself first...

[fredstro]

@videlec What do you think about Matthias Koeppes response (https://groups.google.com/g/sage-devel/c/QtfFtjdePcY)?

I am thinking that maybe I should aim at an install through:
sage -i hilbert-modular-group
and have the "standard" pip install requiring the --no-build-isolation flag, i.e.
sage -pip install --no-build-isolation hilbert-modular-group
(or having the SAGE_LIBRARY environment variable set).

[videlec]

It is a solution but not great. Most users won't use version n+1 of sage before one or two years (eg the time that the stable release make its way to Debian is even more than that). Which means that in the mean time they can not use your package.

[fredstro]

I understand that but I don't know of any perfect solution.

The most straightforward solution would be to go against PEP517 and delete pyproject.toml so pip uses the legacy build backend without build isolation. This would probably work for at least a few years until the pip developers deprecate it.

Another more complicated option might be to setup some kind of buildbot to make wheels for as many platforms as possible. Since pip prefers installs from existing wheels before source distributions that should make it possible for most users to just do sage -pip install without the --no-build-isolation flag.