BLD: use meson's NumPy resolution mechanism for building

[jorisvandenbossche]

Continuation of #61095

[rgommers]

The PKG_CONFIG_PATH usages still stand out to me as most likely unnecessary (see #61095 (comment)).

In case they are needed for a cross-compile job (do you have any here?), then there is no need to poke at private numpy internals, you can get the right path by simply running:

$ numpy-config --pkgconfigdir

[jorisvandenbossche]

Yes, I was going to post some follow-up questions/notes after reading through the whole discussion in the original PR:

• My understanding is that PKG_CONFIG_PATH is only needed in case of cross-compiling, and thus can be removed in most places (eg most units tests or the doc build workflow is not doing any cross-compilation). You confirmed that.

• AFAIK (and from checking some logs), we do have cross-compilation in our CI, but only for pyodide (that's also the only wheel build that is failing, since the setting of the env variable is not yet working properly). All the other builds seem to run natively on custom images or containers.

  But one thing I don't entirely understand is what path to set for that environment variable (probably due to lack of any actual experience with cross-compilation up to now ;)). I would expect that we have to set the path for the numpy on the host machine? (since the problem with cross-compilation is that running python code to import numpy to get the include path on the build machine gives the wrong info, and running python code with the host python is not possible. Of course, with this new meson dependency(), that avoids running python code, but the info still needs to be correct for the host python?)
  But the current way in the PR (PKG_CONFIG_PATH=$(python -c 'import site; print(site.getsitepackages()[0])')/numpy/_core/lib/pkgconfig uses python on the build machine? And also numpy-config --pkgconfigdir does that? Although I see that is also essentially what scipy is doing in the conda-forge recipe (https://github.com/conda-forge/scipy-feedstock/blob/f685b94150920ec1bdb44caf0eb6b82e4e795600/recipe/build.sh#L11), so I assume I just misunderstanding here.

[rgommers]

Oh you're not wrong, having to run the host interpreter is pretty much the most critical point for cross compilation. That said, conda-forge relies on crossenv, which is basically a set of hacks to make a native interpreter look like a host one. So you can still run it on the build machine.

The other way to deal with it is a cross file, which is typically being written out by distro tooling for things like compilers; passing a second cross file with a numpy path may be needed depending on the exact setup. In most projects I've kept the np.get_include() stanza, to avoid having to deal with that in the not-yet-covered cases.

pyodide-build should already know how to deal with numpy, e.g. https://github.com/pyodide/pyodide-build/blob/c0cf55fea19662daa5f41a9fd2da38986742c5c7/pyodide_build/config.py#L318. I can't quite remember how though. @agriyakhetarpal maybe you could weigh in here on the specifics for the Pyodide job here?

[agriyakhetarpal]

Hi @rgommers and @jorisvandenbossche, you are right that the environment variable doesn't work in this case. I noticed that the pkg-config step in the Pyodide job fails silently:

⏵ Run echo "PKG_CONFIG_PATH=$(numpy-config --pkgconfigdir)" >> $GITHUB_ENV
/home/runner/work/_temp/3cf03974-67a6-46da-ac22-61149010c06b.sh: line 1: numpy-config: command not found

I think this links back to this draft PR in pyodide-build: pyodide/pyodide-build#21. The problem is that when a build-time dependency is installed, its declared console scripts, in this case, numpy-config, are not exposed. This is the same problem as the one we have for SciPy's builds, where the custom f2py generators need to be patched out. Since it does not matter for the build to proceed, one less-than-ideal workaround is to install NumPy in the build environment beforehand (as you would when building without build-time isolation) so that numpy-config is available on PATH.

I'll note that we do ship a Meson cross file; accessible via the command line as pyodide config get meson_cross_file or via an environment variable as MESON_CROSS_FILE, and overrideable when set as a value under the [tool.pyodide.build] table in pyproject.toml or from the environment variable).

[Alvaro-Kothe]

If you still intend to support numpy<2, you can create a NumPy dependency as it was done before in incdir_numpy.

Suggested change

	numpy_dep = dependency('numpy', method: 'auto')
	numpy_dep = dependency('numpy', required: false)
	if not numpy_dep.found()
	incdir_numpy = run_command(
	py,
	[
	'-c',
	'''
	import os
	import numpy as np
	try:
	# Check if include directory is inside the pandas dir
	# e.g. a venv created inside the pandas dir
	# If so, convert it to a relative path
	incdir = os.path.relpath(np.get_include())
	except Exception:
	incdir = np.get_include()
	print(incdir)
	''',
	],
	check: true,
	).stdout().strip()
	numpy_dep = declare_dependency(include_directories: incdir_numpy)
	endif

[jorisvandenbossche]

My understanding is that this is run / checked during build time only, and we already do require numpy>=2 for building (in the build-system.requires key in pyproject.toml), so we can assume numpy>=2 here ?

[Alvaro-Kothe]

In the failing CI jobs it's being used numpy==1.26.0, so it's possible to bypass this requirement with --no-build-isolation.

[jorisvandenbossche]

Ah yes. I had seen that failure and was planning to update that to use a newer numpy or to not use build-isolation, as that feels like an error on our side (as a user of pandas in our own CI), we should honor our own build requirements.

In general if you use build isolation, you are responsible to make sure to have all needed libraries and correct versions installed, so my feeling is that we don't have to account for the use case of older numpy in that case.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.