Skip to content

Conversation

@dimpase
Copy link
Member

@dimpase dimpase commented Oct 3, 2025

re-add sage_numerical_backends_coin into sagelib

📝 Checklist

  • The title is concise and informative.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation and checked the documentation preview.

⌛ Dependencies

@dimpase
Copy link
Member Author

dimpase commented Oct 3, 2025

at the moment the upstream package file is just a temporary file, but I'll make a proper one.

I propose stop linking the version of this package and the Sage version.

@dimpase dimpase mentioned this pull request Oct 3, 2025
5 tasks
@github-actions
Copy link

github-actions bot commented Oct 3, 2025

Documentation preview for this PR (built with commit 360d485; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

@dcoudert
Copy link
Contributor

dcoudert commented Oct 3, 2025

What's the right way to test this PR ?

I tried this and it fails

MAC-04017247:sage dcoudert$ ./sage -pip install sage-numerical-backends-coin
Collecting sage-numerical-backends-coin
  Using cached sage_numerical_backends_coin-10.4.tar.gz (23 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [101 lines of output]
      Collecting setuptools>=68.1.1
        Using cached setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB)
      Collecting pkgconfig
        Using cached pkgconfig-1.5.5-py3-none-any.whl.metadata (4.0 kB)
      Collecting cython!=3.0.3,<4.0,>=3.0
        Using cached cython-3.1.4-cp312-cp312-macosx_11_0_arm64.whl.metadata (5.0 kB)
      Collecting cysignals>=1.10.2
        Using cached cysignals-1.12.5-cp312-cp312-macosx_11_0_arm64.whl.metadata (12 kB)
      Collecting sagemath-environment
        Using cached sagemath_environment-10.7-py3-none-any.whl.metadata (3.1 kB)
      Collecting sagemath-categories
        Using cached sagemath_categories-10.7.tar.gz (563 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'error'
        error: subprocess-exited-with-error
      
        × pip subprocess to install build dependencies did not run successfully.
        │ exit code: 1
        ╰─> [72 lines of output]
            Collecting setuptools>=68.1.1
              Using cached setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB)
            Collecting wheel>=0.36.2
              Using cached wheel-0.45.1-py3-none-any.whl.metadata (2.3 kB)
            Collecting sage-setup~=10.7
              Using cached sage_setup-10.7-py3-none-any.whl.metadata (1.3 kB)
            Collecting sagemath-environment~=10.7
              Using cached sagemath_environment-10.7-py3-none-any.whl.metadata (3.1 kB)
            Collecting sagemath-objects~=10.7
              Using cached sagemath_objects-10.7.tar.gz (667 kB)
              Installing build dependencies: started
              Installing build dependencies: finished with status 'done'
              Getting requirements to build wheel: started
              Getting requirements to build wheel: finished with status 'error'
              error: subprocess-exited-with-error
      
              × Getting requirements to build wheel did not run successfully.
              │ exit code: 1
              ╰─> [43 lines of output]
                  Traceback (most recent call last):
                    File "/opt/homebrew/bin/meson", line 3, in <module>
                      from mesonbuild.mesonmain import main
                  ModuleNotFoundError: No module named 'mesonbuild'
                  ninja: error: rebuilding 'build.ninja': subcommand failed
                  ************************************************************************
                  Traceback (most recent call last):
                    File "/Users/dcoudert/sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 345, in _rebuild
                      subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=subprocess.DEVNULL, check=True)
                    File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 571, in run
                      raise CalledProcessError(retcode, process.args,
                  subprocess.CalledProcessError: Command '['/opt/homebrew/bin/ninja']' returned non-zero exit status 1.
      
                  The above exception was the direct cause of the following exception:
      
                  Traceback (most recent call last):
                    File "/Users/dcoudert/sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
                      main()
                    File "/Users/dcoudert/sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
                      json_out['return_val'] = hook(**hook_input['kwargs'])
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    File "/Users/dcoudert/sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
                      return hook(config_settings)
                             ^^^^^^^^^^^^^^^^^^^^^
                    File "/private/var/folders/hw/w6ldqpgn52vdvv03cgbkkr3h3lhtdp/T/pip-build-env-c6z4uktl/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 331, in get_requires_for_build_wheel
                      return self._get_build_requires(config_settings, requirements=[])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    File "/private/var/folders/hw/w6ldqpgn52vdvv03cgbkkr3h3lhtdp/T/pip-build-env-c6z4uktl/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires
                      self.run_setup()
                    File "/private/var/folders/hw/w6ldqpgn52vdvv03cgbkkr3h3lhtdp/T/pip-build-env-c6z4uktl/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup
                      exec(code, locals())
                    File "<string>", line 21, in <module>
                    File "/private/var/folders/hw/w6ldqpgn52vdvv03cgbkkr3h3lhtdp/T/pip-build-env-c6z4uktl/overlay/lib/python3.12/site-packages/sage_setup/setenv.py", line 17, in setenv
                      from sage.env import SAGE_LOCAL, SAGE_VENV, SAGE_ARCHFLAGS, SAGE_PKG_CONFIG_PATH
                    File "/Users/dcoudert/sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 311, in find_spec
                      tree = self._rebuild()
                             ^^^^^^^^^^^^^^^
                    File "/Users/dcoudert/sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/_sagemath_editable_loader.py", line 347, in _rebuild
                      raise ImportError(f're-building the {self._name} meson-python editable wheel package failed') from exc
                  ImportError: re-building the sagemath meson-python editable wheel package failed
                  ************************************************************************
                  Error building the Sage library
                  ************************************************************************
                  [end of output]
      
              note: This error originates from a subprocess, and is likely not a problem with pip.
            error: subprocess-exited-with-error
      
            × Getting requirements to build wheel did not run successfully.
            │ exit code: 1
            ╰─> See above for output.
      
            note: This error originates from a subprocess, and is likely not a problem with pip.
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: subprocess-exited-with-error
      
      × pip subprocess to install build dependencies did not run successfully.
      │ exit code: 1
      ╰─> See above for output.
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

@dimpase
Copy link
Member Author

dimpase commented Oct 3, 2025

no, just think of it as an optional spkg.

Look up the configure option to use to enable it, run ./configure with this option, then make, etc.

@dcoudert
Copy link
Contributor

dcoudert commented Oct 3, 2025

I can't compile cbc 2.9.4.p0. homebrew version is 2.10.12.

[cbc-2.9.4.p0] [spkg-install] g++ -std=gnu++11 -dynamiclib -single_module  -o .libs/libClp.1.13.6.dylib .libs/libClp.1.13.6.dylib-master.o  -L/Users/dcoudert/sage/local/lib -lreadline -lncurses -L/opt/homebrew/Cellar/coinutils/2.11.12/lib -lCoinUtils -lbz2 -lz -L/opt/homebrew/opt/openblas/lib -lopenblas -lm  -install_name  /Users/dcoudert/sage/local/lib/libClp.1.dylib -compatibility_version 15 -current_version 15.6
[cbc-2.9.4.p0] [spkg-install] ld: warning: -single_module is obsolete
[cbc-2.9.4.p0] [spkg-install] Undefined symbols for architecture arm64:
[cbc-2.9.4.p0] [spkg-install]   "CoinIndexedVector::checkClear()", referenced from:
[cbc-2.9.4.p0] [spkg-install]       ClpDualRowSteepest::updateWeights(CoinIndexedVector*, CoinIndexedVector*, CoinIndexedVector*, CoinIndexedVector*) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpDualRowSteepest::updateWeights(CoinIndexedVector*, CoinIndexedVector*, CoinIndexedVector*, CoinIndexedVector*) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpFactorization::updateColumnForDebug(CoinIndexedVector*, CoinIndexedVector*, bool) const in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ...
[cbc-2.9.4.p0] [spkg-install] ld: symbol(s) not found for architecture arm64
[cbc-2.9.4.p0] [spkg-install] clang++: error: linker command failed with exit code 1 (use -v to see invocation)
[cbc-2.9.4.p0] [spkg-install] make[6]: *** [libClp.la] Error 1
[cbc-2.9.4.p0] [spkg-install] make[5]: *** [all] Error 2
[cbc-2.9.4.p0] [spkg-install] make[4]: *** [all-recursive] Error 1
[cbc-2.9.4.p0] [spkg-install] make[3]: *** [all-recursive] Error 1
[cbc-2.9.4.p0] [spkg-install] ********************************************************************************
[cbc-2.9.4.p0] [spkg-install] Error building cbc-2.9.4.p0
[cbc-2.9.4.p0] [spkg-install] ********************************************************************************

@dimpase
Copy link
Member Author

dimpase commented Oct 3, 2025

I can't compile cbc 2.9.4.p0. homebrew version is 2.10.12.

does it work with with the Homebrew version?

[cbc-2.9.4.p0] [spkg-install] g++ -std=gnu++11 -dynamiclib -single_module  -o .libs/libClp.1.13.6.dylib .libs/libClp.1.13.6.dylib-master.o  -L/Users/dcoudert/sage/local/lib -lreadline -lncurses -L/opt/homebrew/Cellar/coinutils/2.11.12/lib -lCoinUtils -lbz2 -lz -L/opt/homebrew/opt/openblas/lib -lopenblas -lm  -install_name  /Users/dcoudert/sage/local/lib/libClp.1.dylib -compatibility_version 15 -current_version 15.6

either your LDFLAGS are unclean, or cbc's configure finds these coinutils thing which isn't compatible (something I've never seen)

cbc 2.10 has more dependencies, so bumping up the version isn't too easy.

@dcoudert
Copy link
Contributor

dcoudert commented Oct 4, 2025

configure with option --use-system-cbc=yes says

cbc:                            no suitable system package; optional, use "./configure --enable-cbc" to install SPKG version 2.9.4.p0

I tried with option -enable-cbc and then compilation of 2.9.4.p0 fails.

@dimpase
Copy link
Member Author

dimpase commented Oct 5, 2025

OK, I see - cbc depends on openblas, but Homebrew's openblas is incompatible with Sage, as it uses libgomp.dylib, and cannot be used with clang (not without extra hacks).

Namely:

 % pkg-config --cflags openblas
-I/opt/homebrew/Cellar/openblas/0.3.30/include -fopenmp

(and Apple clang doesn't grok -fopenmp, it needs -Xpreprocessor -fopenmp. But it's only part of the story. To use OpenMP with clang, it must be LLVM's OpenMP, not GNU OpenMP - and openblas uses the latter. So it's a mess)

@dimpase dimpase force-pushed the numerical_backends_coin-fix branch from 2baba14 to 03caa14 Compare October 17, 2025 02:39
@dimpase
Copy link
Member Author

dimpase commented Oct 17, 2025

with the latest beta, 10.8.beta7, this much simplified branch works for me on Linux

@dimpase
Copy link
Member Author

dimpase commented Oct 17, 2025

the next step is to update cbc spkg. I am trying the latest version cbc 2.10.12.

It has a number of new dependencies, of which dylp is broken and semi-obsolete, so it has to be configured with --without-dylp, and the same applies to the dependency cgl (version 0.60.9).

I am getting errors in MILP code, e.g. in
src/sage/graphs/generic_graph.py

sage: g = digraphs.RandomDirectedGNP(11, .3)  # reduced from 30 to 11, cf. #32169 ## line 7433 ##
sage: k = Integer(g.edge_connectivity()) ## line 7434 ##
sage: while not k:
....:     g = digraphs.RandomDirectedGNP(11, .3)
....:     k = Integer(g.edge_connectivity()) ## line 7435 ##
python3: CbcModel.cpp:16551: int CbcModel::doOneNode(CbcModel*, CbcNode*&, CbcNode*&): Assertion `masterThread_' failed.
...
[and a coredump follows]

and very similar elsewhere - which seem to indicate that Cbc is multithreaded now, but Sage is not using this option correctly (which is not a surprise).

I'll try do disable threads and see how it goes, but that's certainly not the optimal way to use this software.

@dimpase
Copy link
Member Author

dimpase commented Oct 17, 2025

unfortunately, this leads to an already known rabbit hole, leading to cvxpy spkg, which is in a bad shape, for reasons not really of our only making - see e.g. osqp/qdldl-python#71

@kwankyu
Copy link
Collaborator

kwankyu commented Nov 4, 2025

I propose stop linking the version of this package and the Sage version.

That is sufficient to revive the release github workflow. Done in #41128.

@dimpase dimpase force-pushed the numerical_backends_coin-fix branch from 03caa14 to 360d485 Compare November 7, 2025 19:37
@dimpase
Copy link
Member Author

dimpase commented Nov 7, 2025

@dcoudert - this fixes everything related to cbc - provided you have an external cbc accepted by Sage's configure. We re-incorporate coin_backend into sage and build it using meson.

We should do it conditionally - depending on presence of cbc dependency. This is not implemented yet,
but should be a small change in src/sage/numerical/backends/meson.build.

E.g. you can build one using coinbrew as explained in Readme of https://github.com/coin-or/Cbc.

wget https://raw.githubusercontent.com/coin-or/coinbrew/master/coinbrew
chmod u+x coinbrew
./coinbrew fetch Cbc@master
./coinbrew build Cbc

It should also work on Conda.

To fix the threading issue I for the time being commented out the multiprocessing invocation.

@dimpase
Copy link
Member Author

dimpase commented Nov 7, 2025

We should do it conditionally - depending on presence of cbc dependency. This is not implemented yet,
but should be a small change in src/sage/numerical/backends/meson.build.

@tobiasdiez - is disabler: true in cbc = dependency(...) sufficient to prevent py.extension_module() with
cbc mentioned in dependencies from being (unsuccessfully) built?

@dimpase
Copy link
Member Author

dimpase commented Nov 7, 2025

@tobiasdiez - how does one make sure that Cbc (more precisely, https://anaconda.org/conda-forge/coin-or-cbc )
is undestood by grayskull?
(does one need a PR to change grayskull/grayskull/strategy/py_toml.py?)

override_options: ['cython_language=cpp'],
include_directories: [inc_cpython, inc_rings],
dependencies: [py_dep, cysignals, gmp],
dependencies: [py_dep, cysignals, gmp, cbc],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please extract a deps = [py_dep, cysignals, gmp] variable, and then add cbc conditionally only for the coin_backend (see similar constructions in other meson files)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

coin_backend doesn't need gmp, in fact. Should it just have dependencies: [py_dep, cysignals, cbc], ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in that case add gmp conditionally only for the other backend. In dummy code:

deps = [py_dep, cysignals]
if coin:
  deps += cbc
else
  deps += gmp

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do soon.

sage: COIN()._is_present() # optional - sage_numerical_backends_coin
FeatureTestResult('sage_numerical_backends_coin', True)
"""
JoinFeature.__init__(self, 'sage_numerical_backends_coin',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it also makes sense to rename the feature now to cbc (if it's not used in too many places)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, I didn't rename the feature yet just to make the diff more readable. Will do

@tobiasdiez
Copy link
Contributor

We should do it conditionally - depending on presence of cbc dependency. This is not implemented yet,
but should be a small change in src/sage/numerical/backends/meson.build.

@tobiasdiez - is disabler: true in cbc = dependency(...) sufficient to prevent py.extension_module() with cbc mentioned in dependencies from being (unsuccessfully) built?

yes, that's the purpose of disabler: true

@dimpase dimpase force-pushed the numerical_backends_coin-fix branch from 3732a4b to 0758045 Compare November 8, 2025 04:42
@dimpase
Copy link
Member Author

dimpase commented Nov 8, 2025

@mantepse - have you ever tried src/sage/combinat/bijectionist.py with a non-default ILP solver - in this case Coin-cbc. I see 5 test failures, some of them definitely giving different maths results, not just something like different format of the same thing.

@mantepse
Copy link
Contributor

mantepse commented Nov 8, 2025

@mantepse - have you ever tried src/sage/combinat/bijectionist.py with a non-default ILP solver - in this case Coin-cbc. I see 5 test failures, some of them definitely giving different maths results, not just something like different format of the same thing.

Yes, they used to work at least with scip - but making them work involved quite a bit of work.

@mantepse
Copy link
Contributor

mantepse commented Nov 8, 2025

could you post the failures?

@mantepse
Copy link
Contributor

mantepse commented Nov 8, 2025

One thing the bijectionist uses is to add constraints temporarily.

@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

could you post the failures?

one of them is related to the last commit here. * changed the test for [] to one for length 0, otherwise it prints a huge meaningless list on failure - happening with Cbc

@mantepse
Copy link
Contributor

mantepse commented Nov 9, 2025

Do I understand correctly that the length of the list is 217 with cbc?

@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

Do I understand correctly that the length of the list is 217 with cbc?

yes. and here are the other failures on my machine (the latest cbc, 2.10.12):


src/bin/sage -t --warn-long 5.0 --random-seed=6129053336511794336392055413907032886 src/sage/combinat/bijectionist.py
**********************************************************************
File "src/sage/combinat/bijectionist.py", line 162, in sage.combinat.bijectionist
Failed example:
    len(list(bij.solutions_iterator()))
Expected:
    0
Got:
    217
**********************************************************************
File "src/sage/combinat/bijectionist.py", line 1916, in sage.combinat.bijectionist.Bijectionist._find_counterexample
Failed example:
    next(bij.solutions_iterator())
Expected:
    {'a': 1, 'b': 1, 'c': 2, 'd': 3, 'e': 2}
Got:
    {'a': 2, 'b': 2, 'c': 3, 'd': 1, 'e': 1}
**********************************************************************
File "src/sage/combinat/bijectionist.py", line 1921, in sage.combinat.bijectionist.Bijectionist._find_counterexample
Failed example:
    bij._find_counterexample(bij._A, s0, d, False)
Expected:
    {'a': 2, 'b': 2, 'c': 1, 'd': 3, 'e': 1}
Got:
    {'a': 2, 'b': 2, 'c': 3, 'd': 1, 'e': 1}
**********************************************************************
File "src/sage/combinat/bijectionist.py", line 3188, in sage.combinat.bijectionist._non_copying_intersection.sets.result
Failed example:
    for p in P:
        print(p)
Expected:
    [[1], [1, 2], [1, 2, 3], [1, 2, 3, 4], [1, 2, 3, 4, 5]]
    [[2, 1], [1, 3, 2], [2, 1, 3], [2, 3, 1], [3, 2, 1], ...
    [[3, 1, 2], [1, 4, 2, 3], [2, 4, 1, 3], [3, 1, 2, 4], [3, 1, 4, 2], ...
    [[4, 1, 2, 3], [1, 5, 2, 3, 4], [4, 1, 2, 3, 5], [4, 5, 1, 2, 3], [5, 1, 2, 4, 3], ...
    [[1, 3, 2, 5, 4], [2, 1, 3, 5, 4], [2, 1, 4, 3, 5], [5, 2, 4, 3, 1], [5, 3, 2, 4, 1]]
    [[1, 3, 5, 2, 4], [2, 4, 1, 3, 5], [3, 5, 2, 4, 1], [4, 1, 3, 5, 2], [5, 2, 4, 1, 3]]
    ...
Got:
    [[1], [1, 2], [1, 2, 3], [1, 2, 3, 4], [1, 2, 3, 4, 5]]
    [[2, 1], [1, 3, 2], [2, 1, 3], [2, 3, 1], [3, 2, 1], [1, 2, 4, 3], [1, 3, 2, 4], [1, 3, 4, 2], [1, 4, 3, 2], [2, 1, 3, 4], [2, 1, 4, 3], [2, 3, 1, 4], [2, 3, 4, 1], [2, 4, 3, 1], [3, 2, 1, 4], [3, 2, 4, 1], [4, 2, 3, 1], [4, 3, 2, 1], [1, 2, 3, 5, 4], [1, 2, 4, 3, 5], [1, 2, 4, 5, 3], [1, 2, 5, 4, 3], [1, 3, 2, 4, 5], [1, 3, 4, 2, 5], [1, 3, 4, 5, 2], [1, 3, 5, 4, 2], [1, 4, 3, 2, 5], [1, 4, 3, 5, 2], [1, 5, 3, 4, 2], [2, 1, 3, 4, 5], [2, 3, 1, 4, 5], [2, 3, 4, 1, 5], [2, 3, 4, 5, 1], [2, 3, 5, 4, 1], [2, 4, 3, 1, 5], [2, 4, 3, 5, 1], [2, 5, 3, 4, 1], [3, 2, 1, 4, 5], [3, 2, 4, 1, 5], [3, 2, 4, 5, 1], [3, 2, 5, 4, 1], [4, 2, 3, 1, 5], [4, 2, 3, 5, 1], [5, 2, 3, 4, 1]]
    [[3, 1, 2], [1, 4, 2, 3], [2, 4, 1, 3], [3, 1, 2, 4], [3, 1, 4, 2], [3, 4, 1, 2], [3, 4, 2, 1], [4, 1, 3, 2], [4, 2, 1, 3], [4, 3, 1, 2], [1, 2, 5, 3, 4], [1, 4, 2, 3, 5], [1, 5, 2, 4, 3], [1, 5, 3, 2, 4], [3, 1, 2, 4, 5], [3, 4, 5, 1, 2], [4, 1, 3, 2, 5], [4, 2, 1, 3, 5], [5, 1, 3, 4, 2], [5, 2, 1, 4, 3], [5, 2, 3, 1, 4]]
    [[4, 1, 2, 3], [1, 5, 2, 3, 4], [4, 1, 2, 3, 5], [5, 1, 2, 4, 3], [5, 1, 3, 2, 4], [5, 2, 1, 3, 4]]
    [[1, 3, 2, 5, 4], [2, 1, 3, 5, 4], [2, 1, 4, 3, 5], [5, 2, 4, 3, 1], [5, 3, 2, 4, 1]]
    [[1, 3, 5, 2, 4], [2, 4, 1, 3, 5], [3, 5, 2, 4, 1], [4, 1, 3, 5, 2], [5, 2, 4, 1, 3]]
    [[1, 4, 2, 5, 3], [2, 5, 3, 1, 4], [3, 1, 4, 2, 5], [4, 2, 5, 3, 1], [5, 3, 1, 4, 2]]
    [[1, 4, 5, 2, 3], [3, 4, 1, 2, 5], [3, 5, 1, 4, 2], [4, 2, 5, 1, 3], [4, 5, 3, 1, 2]]
    [[1, 4, 5, 3, 2], [3, 1, 5, 4, 2], [3, 2, 5, 1, 4], [3, 4, 2, 1, 5], [5, 4, 3, 1, 2]]
    [[1, 5, 4, 2, 3], [2, 5, 1, 4, 3], [4, 2, 1, 5, 3], [4, 3, 1, 2, 5], [4, 5, 3, 2, 1]]
    [[1, 5, 4, 3, 2], [2, 1, 5, 4, 3], [3, 2, 1, 5, 4], [4, 3, 2, 1, 5], [5, 4, 3, 2, 1]]
    [[2, 1, 4, 5, 3], [2, 3, 1, 5, 4], [2, 5, 4, 3, 1], [4, 3, 2, 5, 1], [5, 3, 4, 2, 1]]
    [[2, 1, 5, 3, 4], [3, 1, 2, 5, 4], [5, 1, 4, 3, 2], [5, 3, 2, 1, 4], [5, 4, 2, 3, 1]]
    [[2, 3, 5, 1, 4], [2, 4, 5, 3, 1], [3, 1, 4, 5, 2], [3, 4, 2, 5, 1], [5, 3, 4, 1, 2]]
    [[2, 4, 1, 5, 3], [2, 5, 4, 1, 3], [3, 5, 4, 2, 1], [4, 3, 1, 5, 2], [4, 3, 5, 2, 1]]
    [[2, 4, 5, 1, 3], [3, 4, 1, 5, 2], [3, 4, 5, 2, 1], [3, 5, 4, 1, 2], [4, 3, 5, 1, 2]]
    [[2, 5, 1, 3, 4], [4, 1, 2, 5, 3], [4, 5, 2, 3, 1], [5, 1, 4, 2, 3], [5, 3, 1, 2, 4]]
    [[3, 1, 5, 2, 4], [3, 5, 2, 1, 4], [4, 1, 5, 3, 2], [5, 4, 1, 3, 2], [5, 4, 2, 1, 3]]
    [[3, 5, 1, 2, 4], [4, 1, 5, 2, 3], [4, 5, 1, 3, 2], [4, 5, 2, 1, 3], [5, 4, 1, 2, 3]]
**********************************************************************
File "src/sage/combinat/bijectionist.py", line 3198, in sage.combinat.bijectionist._non_copying_intersection.sets.result
Failed example:
    for d in sorted(bij.minimal_subdistributions_blocks_iterator(), key=lambda d: (len(d[0]), d[0])):
        print(d)
Exception raised:
    Traceback (most recent call last):
      File "/home/dima/software/sage/src/sage/doctest/forker.py", line 734, in _run
        self.compile_and_execute(example, compiler, test.globs)
        ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/dima/software/sage/src/sage/doctest/forker.py", line 1158, in compile_and_execute
        exec(compiled, globs)
        ~~~~^^^^^^^^^^^^^^^^^
      File "<doctest sage.combinat.bijectionist._non_copying_intersection.sets.result[35]>", line 1, in <module>
        for d in sorted(bij.minimal_subdistributions_blocks_iterator(), key=lambda d: (len(d[Integer(0)]), d[Integer(0)])):
                 ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/dima/software/sage/src/sage/combinat/bijectionist.py", line 2105, in minimal_subdistributions_blocks_iterator
        add_counter_example_constraint(s)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
      File "/home/dima/software/sage/src/sage/combinat/bijectionist.py", line 2078, in add_counter_example_constraint
        minimal_subdistribution.add_constraint(sum(D[p] for p in P
                                               ~~~^^^^^^^^^^^^^^^^
                                                   if s[p] == v) == V[v])
                                                   ^^^^^^^^^^^^^
      File "/home/dima/software/sage/src/sage/combinat/bijectionist.py", line 2079, in <genexpr>
        if s[p] == v) == V[v])
           ~^^^
    KeyError: [3, 1, 2]
**********************************************************************
3 items had failures:
   1 of 101 in sage.combinat.bijectionist
   2 of  10 in sage.combinat.bijectionist.Bijectionist._find_counterexample
   2 of  37 in sage.combinat.bijectionist._non_copying_intersection.sets.result
    [588 tests, 5 failures, 33.26s wall]
----------------------------------------------------------------------
src/bin/sage -t --warn-long 5.0 --random-seed=6129053336511794336392055413907032886 src/sage/combinat/bijectionist.py  # 5 doctests failed
----------------------------------------------------------------------
Total time for all tests: 33.3 seconds
    cpu time: 33.1 seconds
    cumulative wall time: 33.3 seconds
Features detected for doctesting: sage.numerical.mip

@dimpase dimpase changed the title sage_numerical_backends_coin and external cython fix sage_numerical_backends_coin fix Nov 9, 2025
@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

@tobiasdiez - what's the role of tools/update-meson.py ? I noticed that for the 1st attempt at commit

@dimpase
disable threading in cbc - fix is WIP
360d485

I got an action error with something about updated meson file. The error output showed the diff of the changed meson.build w.r.t. to the previous version.

@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

regarding cbc spkg, I propose to convert it into a dummy package, like we did with r.
Otherwise we'd need to add half a dozen new optional spkgs: C/C++ libs, and test data packages.

Cbc is easy to install using their preferred way (and many distros provide it).
If it's not installed into a standard location, one would need to add their lib/pkgconfig directory to PKG_CONFIG_PATH. We can also make ./configure do it - --enable-config=<prefix> can be made to do this.

@mantepse
Copy link
Contributor

mantepse commented Nov 9, 2025

I ran the tests with SCIP, all but two tests, including the long ones, pass. The two which don't pass are

File "src/sage/combinat/bijectionist.py", line 1917, in sage.combinat.bijectionist.Bijectionist._find_counterexample
Failed example:
    next(bij.solutions_iterator())
Expected:
    {'a': 1, 'b': 1, 'c': 2, 'd': 3, 'e': 2}
Got:
    {'a': 2, 'b': 2, 'c': 1, 'd': 1, 'e': 3}
**********************************************************************
File "src/sage/combinat/bijectionist.py", line 1922, in sage.combinat.bijectionist.Bijectionist._find_counterexample
Failed example:
    bij._find_counterexample(bij._A, s0, d, False)
Expected:
    {'a': 2, 'b': 2, 'c': 1, 'd': 3, 'e': 1}
Got:
    {'a': 2, 'b': 2, 'c': 1, 'd': 1, 'e': 3}

These are differences which are mathematically OK. Of course, the tests could possibly be improved.

@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

these tests are easy to improve - just collect the known outcomes for different solvers in a list, and test that the result is in the list.

@dcoudert
Copy link
Contributor

dcoudert commented Nov 9, 2025

I installed cbc using coinbrew as explained above.
However, configure don't care as homebrew openblas is not accepted.

Checking whether SageMath should install SPKG cbc...
checking whether any of openblas is installed as or will be installed as SPKG... yes; install cbc as well
real_configure: no suitable system package found for SPKG cbc

and then make fails trying to compile cbc 2.9.4.p0

[cbc-2.9.4.p0] [spkg-install] g++ -std=gnu++11 -dynamiclib -single_module  -o .libs/libClp.1.13.6.dylib .libs/libClp.1.13.6.dylib-master.o  -L/Users/dcoudert/sage/local/lib -lreadline -lncurses -L/opt/homebrew/Cellar/coinutils/2.11.12/lib -lCoinUtils -lbz2 -lz -L/opt/homebrew/opt/openblas/lib -lopenblas -lm  -install_name  /Users/dcoudert/sage/local/lib/libClp.1.dylib -compatibility_version 15 -current_version 15.6
[cbc-2.9.4.p0] [spkg-install] ld: warning: -single_module is obsolete
[cbc-2.9.4.p0] [spkg-install] Undefined symbols for architecture arm64:
[cbc-2.9.4.p0] [spkg-install]   "CoinIndexedVector::checkClear()", referenced from:
[cbc-2.9.4.p0] [spkg-install]       ClpDualRowSteepest::updateWeights(CoinIndexedVector*, CoinIndexedVector*, CoinIndexedVector*, CoinIndexedVector*) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpDualRowSteepest::updateWeights(CoinIndexedVector*, CoinIndexedVector*, CoinIndexedVector*, CoinIndexedVector*) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpFactorization::updateColumnForDebug(CoinIndexedVector*, CoinIndexedVector*, bool) const in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ClpSimplex::createRim(int, bool, int) in libClp.1.13.6.dylib-master.o
[cbc-2.9.4.p0] [spkg-install]       ...
[cbc-2.9.4.p0] [spkg-install] ld: symbol(s) not found for architecture arm64
[cbc-2.9.4.p0] [spkg-install] clang++: error: linker command failed with exit code 1 (use -v to see invocation)
[cbc-2.9.4.p0] [spkg-install] make[8]: *** [libClp.la] Error 1
[cbc-2.9.4.p0] [spkg-install] make[7]: *** [all] Error 2
[cbc-2.9.4.p0] [spkg-install] make[6]: *** [all-recursive] Error 1
[cbc-2.9.4.p0] [spkg-install] make[5]: *** [all-recursive] Error 1
[cbc-2.9.4.p0] [spkg-install] ********************************************************************************
[cbc-2.9.4.p0] [spkg-install] Error building cbc-2.9.4.p0
[cbc-2.9.4.p0] [spkg-install] ********************************************************************************

@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

you need to put the location of coinbrew-installed cbc into PKG_CONFIG_PATH (make sure it comes before homebrew one)

@dcoudert
Copy link
Contributor

dcoudert commented Nov 9, 2025

you need to put the location of coinbrew-installed cbc into PKG_CONFIG_PATH (make sure it comes before homebrew one)

I tried with export PKG_CONFIG_PATH=$HOME/Applications/coin-or:$HOME/Applications/coin-or/Cbc:$HOME/Applications/coin-or/dist:$HOME/Applications/coin-or/dist/bin:$PKG_CONFIG_PATH (I'm not sure which path is needed) but it's not sufficient.
Should I uninstall homebrew openblas ?

To ensure that it's before homebrew once, I have to do the export after bootstrap and source .homebrew... and before configure....

@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

one should run source .homebrew before ./bootstrap. anyhow, you can check what's found by pkg-config under ./sage --buildsh shell, by running pkg-config --path cbc.
(this is the location picked up by ./configure)

This should return <your coinbrew directory>/dist/lib/pkgconfig/cbc.pc.

@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

@dcoudert - you need PKG_CONFIG_PATH=$HOME/Applications/coin-or/dist/lib/pkgconfig:....

@dimpase
Copy link
Member Author

dimpase commented Nov 9, 2025

@dcoudert is your coinbrew cbc actually dependent on openblas?
You can see it in the generated cbc.pc and by running otool -L on the built libcbc.dylib

Same question for the homebrew one (it's declared to be dependent on openblas, but is it actually picked up during the build?)

@dcoudert
Copy link
Contributor

dcoudert commented Nov 9, 2025

no dependency on openblas when using the coinbrew version, but there is one with the hombre version.

Content of the file cbc.pc when building with coinbrew

MAC-04017247:dist dcoudert$ m lib/pkgconfig/cbc.pc 
prefix=/Users/dcoudert/Applications/coin-or/dist
#prefix=${pcfiledir}/../..
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include/coin-or

Name: Cbc
Description: COIN-OR Branch and Cut
URL: https://github.com/coin-or/Cbc
Version: devel
Cflags: -I${includedir}
Libs: -L${libdir} -lCbc
Requires.private: osi-clp clp cgl osi coinutils 
#Libs: -L${libdir} -lCbc -lpthread -L/Users/dcoudert/Applications/coin-or/dist/lib -lCgl -lOsiClp -lClp -lOsi -lCoinUtils -lreadline -lncurses       
#Requires: osi-clp clp cgl osi coinutils 

and the output of otool -L libCbc.dylib

lib/libCbc.dylib:
	/Users/dcoudert/Applications/coin-or/dist/lib/libCbc.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1356.0.0)
	/Users/dcoudert/Applications/coin-or/dist/lib/libCgl.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/dcoudert/Applications/coin-or/dist/lib/libOsiClp.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/dcoudert/Applications/coin-or/dist/lib/libClp.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/dcoudert/Applications/coin-or/dist/lib/libOsi.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/dcoudert/Applications/coin-or/dist/lib/libCoinUtils.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.8)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.12)
	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 2000.63.0)

Now for the homebrew version:
Content of the file cbc.pc

prefix=/opt/homebrew/Cellar/cbc/2.10.12
exec_prefix=${prefix}
libdir=/opt/homebrew/Cellar/cbc/2.10.12/lib
includedir=/opt/homebrew/Cellar/cbc/2.10.12/include/cbc/coin

Name: CBC
Description: COIN-OR Branch and Cut
URL: https://projects.coin-or.org/Cbc
Version: 2.10.12
Libs: -L${libdir} -lCbcSolver -lCbc -lpthread 
Cflags: -I${includedir}
Requires: clp osi-clp cgl osi coinutils 

and here we get a dependency on openblas

libCbc.dylib:
	/opt/homebrew/opt/cbc/lib/libCbc.3.dylib (compatibility version 14.0.0, current version 14.12.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)
	/opt/homebrew/opt/cgl/lib/libCgl.1.dylib (compatibility version 12.0.0, current version 12.9.0)
	/opt/homebrew/opt/clp/lib/libOsiClp.1.dylib (compatibility version 16.0.0, current version 16.10.0)
	/opt/homebrew/opt/clp/lib/libClpSolver.1.dylib (compatibility version 16.0.0, current version 16.10.0)
	/opt/homebrew/opt/clp/lib/libClp.1.dylib (compatibility version 16.0.0, current version 16.10.0)
	/opt/homebrew/opt/osi/lib/libOsi.1.dylib (compatibility version 15.0.0, current version 15.11.0)
	/opt/homebrew/opt/coinutils/lib/libCoinUtils.3.dylib (compatibility version 15.0.0, current version 15.12.0)
	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.8)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.12)
	/opt/homebrew/opt/openblas/lib/libopenblas.0.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.0)

@dimpase
Copy link
Member Author

dimpase commented Nov 10, 2025

coinbrew version depends on Accelerate (Apple's blas/lapack).

you may try my experimental branch on #40397 (sagemath with Accelerate instead of openblas on macOS) - I need to update it first

@dcoudert
Copy link
Contributor

I tried to combine with #40397 but it's the same result, even when ensuring that homebrew openblas is not in PKG_CONFIG_PATH. Note that all my trials are in fresh shells.

@dimpase
Copy link
Member Author

dimpase commented Nov 10, 2025

I tried to combine with #40397 but it's the same result, even when ensuring that homebrew openblas is not in PKG_CONFIG_PATH. Note that all my trials are in fresh shells.

I just updated the instructions for #40397 - there is no special step needed any more, I forgot to do this. But this should not affect the result (adding a non-existent path to PKG_CONFIG_PATH is (probably) harmless).

What do you see if you run (after ./bootstrap && ./configure)

./sage --buildsh
pkg-config --path openblas

?

And please attach config.log here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants