Add DIFFENGINE as canonicalization backend by Transurgeon · Pull Request #172 · cvxgrp/DNLP

Transurgeon · 2026-03-04T14:58:18Z

DIFFENGINE uses differentiation (via sparsediffpy) to directly extract concrete A, b, c matrices instead of building parametric tensors through CoeffExtractor. This provides an alternative canonicalization path that can be selected with canon_backend='DIFFENGINE'.

Key changes:

Add DIFFENGINE_CANON_BACKEND constant and early return in get_canon_backend()
Branch in ConeMatrixStuffing.apply() to call build_diffengine_cone_program()
Add DiffengineConeProgram class extending ParamConeProg
Handle DiffengineConeProgram in conic_solver.format_constraints()
Add vstack converter for diff_engine
Update sparsediffpy dependency to >= 0.1.3

Description

Please include a short summary of the change.
Issue link (if applicable):

Type of change

New feature (backwards compatible)
New feature (breaking API changes)
Bug fix
Other (Documentation, CI, ...)

Contribution checklist

Add our license to new files.
Check that your code adheres to our coding style.
Write unittests.
Run the unittests and check that they’re passing.
Run the benchmarks to make sure your change doesn’t introduce a regression.

initial attempts at adding a smooth canon for maximum

Ipopt interface prototype

adding more smooth canonicalizers

* adds oracles and bounds class to ipopt interface * adds some settings and solver lists changes for IPOPT * adds nlp solver option and can call ipopt * adds more experiments for integrating ipopt as a solver interface * passing the problem through the inversion * add some more extra changes * adding nlmatrixstuffing --------- Co-authored-by: William Zijie Zhang <william@gridmatic.com>

Co-authored-by: William Zijie Zhang <william@gridmatic.com>

* adding many tests, new smoothcanon for min, and improvements to ipopt_nlpif * fixing last two tests * add another example, qcp * adding example for acopf * add control of a car example done --------- Co-authored-by: William Zijie Zhang <william@gridmatic.com>

* update solution statuses thanks to odow * removes unusued solver information --------- Co-authored-by: William Zijie Zhang <william@gridmatic.com>

* getting rocket landing example to work * add changes to the jacobian computation --------- Co-authored-by: William Zijie Zhang <william@gridmatic.com>

* adding many more example of non-convex functions * making lots of progress on understanding good canonicalizations --------- Co-authored-by: William Zijie Zhang <william@gridmatic.com>

Co-authored-by: William Zijie Zhang <william@gridmatic.com>

* Rename ESR/HSR to linearizable_convex/linearizable_concave Spell out opaque acronyms for clarity per PR review feedback: is_atom_esr → is_atom_linearizable_convex, is_atom_hsr → is_atom_linearizable_concave, is_esr → is_linearizable_convex, is_hsr → is_linearizable_concave, is_smooth → is_linearizable. Docstrings clarify that "linearizable convex" means the expression is convex after linearizing all smooth subexpressions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Adopt three-way atom classification: smooth, nonsmooth-convex, nonsmooth-concave Replace the two-axis is_atom_linearizable_convex/concave overrides across all atoms with a single method from the paper's three categories: is_atom_smooth, is_atom_nonsmooth_convex, or is_atom_nonsmooth_concave. The base class derives the old linearizable methods for backward compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Inline atom-level linearizable checks into expression-level composition rules Remove the intermediate is_atom_linearizable_convex/concave methods and _has_dnlp_classification helper, which were only used in the two expression-level composition rules in atom.py. The three-way classification (smooth, nonsmooth-convex, nonsmooth-concave) is now used directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Simplify is_atom_smooth docstring Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Propagate initial values to reduced variables in CvxAttr2Constr When CvxAttr2Constr creates reduced variables for dim-reducing attributes (e.g., diag), the original variable's initial value was not being lowered and assigned to the reduced variable. This caused NLP solvers to fail with "Variable has no value" during initial point construction. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Handle sparse values when propagating diag variable initials When a diag variable has its value set as a sparse matrix, extract the diagonal directly via scipy rather than passing it through np.diag which does not support sparse inputs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Revert "Handle sparse values when propagating diag variable initials" This reverts commit e422565. * Revert "Propagate initial values to reduced variables in CvxAttr2Constr" This reverts commit d02a758. * fix pnorm nonsmooth convex --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

When CvxAttr2Constr creates reduced variables for dimension-reducing attributes (e.g. diag=True), the initial value was not propagated to the reduced variable, causing NLP solvers to fail. This mirrors the existing value propagation already done for parameters. Also handle sparse diagonal matrices in lower_value() by using .diagonal() instead of np.diag() which doesn't accept sparse input. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

#153) * Make is_linearizable_convex/is_linearizable_concave abstract on Expression Enforce implementation in all Expression subclasses by uncommenting @abc.abstractmethod decorators. Add missing implementations to indicator (convex, not concave) and PartialProblem (delegates to is_convex/is_concave). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * remove is_smooth from max --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Separates test-only utility from production code by moving it to cvxpy/tests/nlp_tests/derivative_checker.py and updating all 19 test file imports. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Address review comments: remove redundant nonsmooth methods, add types, docs - Remove is_atom_nonsmooth_convex/is_atom_nonsmooth_concave from base Atom class and 10 atom subclasses; use existing is_atom_convex/is_atom_concave in DNLP composition rules instead - Add type annotations and docstrings to NLPsolver, Bounds, and Oracles in nlp_solver.py - Document Variable.sample_bounds with class-level type annotation and docstring - Revert GENERAL_PROJECTION_TOL back to 1e-10 (was loosened for removed IPOPT derivative checker) - Update CLAUDE.md atom classification docs to reflect simplified API Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move sample_bounds docs from #: comments into Variable class docstring Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…156) * Extract NLP solving logic from problem.py into nlp_solving_chain.py Move the ~85-line NLP block from Problem._solve() and the two initial point methods into a dedicated module. This addresses PR review feedback: - NLP chain building, initial point logic, and solve orchestration now live in cvxpy/reductions/solvers/nlp_solving_chain.py - Use var.get_bounds() instead of var.bounds so sign attributes (nonneg, nonpos) are incorporated into bounds automatically - Initial point helpers are now private module-level functions instead of public Problem methods Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update NLP initial point tests to use _set_nlp_initial_point Tests now import and call the module-level helper directly instead of the removed Problem.set_NLP_initial_point() method. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove debug print and expand comment in best_of loop Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Gate best_of print on verbose flag Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address PR review comments on nlp_solving_chain extraction - Add circular import comment in problem.py explaining the deferred import - Move NLP_SOLVER_VARIANTS from nlp_solving_chain.py to defines.py - Set BOUNDED_VARIABLES = True on NLPsolver base class and use it in _build_nlp_chain (matching the conic solver pattern in solving_chain.py) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * adds changes to test_problem with parametrize * Revert verbose unpack comment to original single-line version Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…160) * uses new coo handling * added comment * fix pre-commit * fix error message if sample bounds are not set (#161)

* Remove conda from CI, use uv + system IPOPT Replace conda-based test_nlp_solvers workflow with uv, installing IPOPT via system packages (apt on Ubuntu, brew on macOS) instead of conda-forge. Uncomment IPOPT optional dependency in pyproject.toml so uv sync --extra IPOPT works. Update installation docs in CLAUDE.md and README.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Keep IPOPT extra commented out to avoid --all-extras breakage The test_optional_solvers workflow uses uv sync --all-extras, which would try to build cyipopt without system IPOPT installed. Instead, install cyipopt directly via uv pip in test_nlp_solvers where the system library is available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove UV_SYSTEM_PYTHON to fix externally-managed env error uv pip install fails on Ubuntu when UV_SYSTEM_PYTHON=1 because the system Python is externally managed. Removing it lets uv use its own venv created by uv sync. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Install LAPACK and BLAS dev libraries for cyipopt build on Ubuntu cyipopt links against LAPACK and BLAS which are not installed by default on Ubuntu runners. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use uv venv + uv pip install instead of uv sync for NLP workflow uv sync manages a locked environment that doesn't play well with uv pip install for additional packages. Switch to uv venv + uv pip install to manage the environment directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Includes upstream fixes for quad_form canonicalization, gradient handling, LDL factorization optimization, CI updates, and documentation improvements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* uses new coo handling * added comment * fix pre-commit * fix error message if sample bounds are not set * docs for DNLP * remove whitespace * split table up into two

* uses new coo handling * added comment * fix pre-commit * fix error message if sample bounds are not set * docs for DNLP * remove whitespace * split table up into two * cache C problem in best of

- Remove `submodules: recursive` from CI workflows (build, test_nlp_solvers, test_backends) - Revert `_grad` in affine_atom.py to upstream version - Remove NaN-allowance block in leaf.py added for NLP structural jacobian - Update is_dnlp() docstrings to use linearizable terminology - Add normalize_shape() helper in converters.py, replacing 7 inline occurrences - Remove unnecessary comment in nlp_solver.py - Export UNO in cvxpy/__init__.py - Revert README.md to upstream CVXPY version Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

#168) * remove unnecessary point in dom + clean up C_problem + clean up oracle * fix william's comments

…nology (#170) - Revert whitespace-only changes in huber, abs, ceil, affine_atom, binary_operators - Revert cosmetic rename and _jacobian_operator addition in index.py - Revert isnan check in leaf.py back to original error handling - Restore removed blank line in quad_form.py - Update DNLP constraint docstrings to use "linearizable convex/concave" terminology Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

- Remove unused `self.broadcast_type` attribute from `broadcast_to.__init__` (was previously used for derivative computation, now handled elsewhere) - Revert CLAUDE.md to upstream master version (DNLP-specific changes were broader than intended for the PR) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

DIFFENGINE uses automatic differentiation (via sparsediffpy) to directly extract concrete A, b, c matrices instead of building parametric tensors through CoeffExtractor. This provides an alternative canonicalization path that can be selected with `canon_backend='DIFFENGINE'`. Key changes: - Add DIFFENGINE_CANON_BACKEND constant and early return in get_canon_backend() - Branch in ConeMatrixStuffing.apply() to call build_diffengine_cone_program() - Add DiffengineConeProgram class extending ParamConeProg - Handle DiffengineConeProgram in conic_solver.format_constraints() - Add vstack converter for diff_engine - Update sparsediffpy dependency to >= 0.1.3 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-03-04T15:04:07Z

Benchmarks that have improved:

   before           after         ratio
 [1cbab94b]       [f3f70f03]

    1.20±0s          1.02±0s     0.85  gini_portfolio.Cajas.time_compile_problem

Benchmarks that have stayed the same:

   before           after         ratio
 [1cbab94b]       [f3f70f03]
     14.3±0ms         15.1±0ms     1.06  simple_LP_benchmarks.SimpleFullyParametrizedLPBenchmark.time_compile_problem
      493±0ms          520±0ms     1.06  semidefinite_programming.SemidefiniteProgramming.time_compile_problem
     15.1±0ms         15.7±0ms     1.04  simple_QP_benchmarks.ParametrizedQPBenchmark.time_compile_problem
      270±0ms          278±0ms     1.03  slow_pruning_1668_benchmark.SlowPruningBenchmark.time_compile_problem
      11.9±0s          12.1±0s     1.02  finance.CVaRBenchmark.time_compile_problem
      740±0ms          755±0ms     1.02  simple_QP_benchmarks.LeastSquares.time_compile_problem
      1.75±0s          1.78±0s     1.01  simple_QP_benchmarks.UnconstrainedQP.time_compile_problem
      989±0ms          999±0ms     1.01  finance.FactorCovarianceModel.time_compile_problem
      1.40±0s          1.40±0s     1.00  matrix_stuffing.ParamConeMatrixStuffing.time_compile_problem
      238±0ms          238±0ms     1.00  simple_QP_benchmarks.SimpleQPBenchmark.time_compile_problem
      134±0ms          134±0ms     1.00  high_dim_convex_plasticity.ConvexPlasticity.time_compile_problem
      280±0ms          281±0ms     1.00  matrix_stuffing.ParamSmallMatrixStuffing.time_compile_problem
      20.5±0s          20.5±0s     1.00  sdp_segfault_1132_benchmark.SDPSegfault1132Benchmark.time_compile_problem
      311±0ms          312±0ms     1.00  gini_portfolio.Yitzhaki.time_compile_problem
      3.88±0s          3.88±0s     1.00  huber_regression.HuberRegression.time_compile_problem
      875±0ms          873±0ms     1.00  simple_LP_benchmarks.SimpleScalarParametrizedLPBenchmark.time_compile_problem
     40.0±0ms         39.9±0ms     1.00  matrix_stuffing.SmallMatrixStuffing.time_compile_problem
      228±0ms          227±0ms     1.00  gini_portfolio.Murray.time_compile_problem
      10.1±0s          10.0±0s     0.99  simple_LP_benchmarks.SimpleLPBenchmark.time_compile_problem
      679±0ms          670±0ms     0.99  matrix_stuffing.ConeMatrixStuffingBench.time_compile_problem
      4.45±0s          4.38±0s     0.99  svm_l1_regularization.SVMWithL1Regularization.time_compile_problem
      5.14±0s          5.05±0s     0.98  optimal_advertising.OptimalAdvertising.time_compile_problem
      1.60±0s          1.57±0s     0.98  tv_inpainting.TvInpainting.time_compile_problem
      2.75±0s          2.67±0s     0.97  quantum_hilbert_matrix.QuantumHilbertMatrix.time_compile_problem

Init derivatives (which allocates dwork scratch space) must happen before forward evaluation, since quad_form's forward() uses dwork for csr_matvec. Also reorder constraint init_jacobian before constraint forward for consistency, and remove redundant problem_init_hessian call. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Extract the DIFFENGINE special-case from ConeMatrixStuffing.apply() and conic_solver.format_constraints() into a clean DiffengineMatrixStuffing sibling class. The solving chain now branches at construction time, and format_constraints uses polymorphic apply_restruct_mat() dispatch. - Extract lower_and_order_constraints() for shared constraint lowering - Add DiffengineMatrixStuffing extending MatrixStuffing (new file) - Add apply_restruct_mat() to ParamConeProg and DiffengineConeProgram - Add SymbolicQuadForm converter for QP path through diffengine - Remove DIFFENGINE branch from ConeMatrixStuffing and isinstance check from conic_solver.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Transurgeon and others added 30 commits June 16, 2025 23:53

adding experiments for ipopt with a cvxpy

bd20969

fixing jacobian sign and putting the constraints in standard form

73b1301

adding some refactoring for automatically calculating bounds

c6adb1a

adds two more examples

00304ba

adding base files for interface

ee8b061

fix rosenbrock with multiple variables

b0d0f19

beginning testing on MLE estimation but encountering errors

205ed3e

fixes test for mle-estimation, thanks parth!

b2fef0c

fixing mle with constraints, but need to generalize

4902202

add new example and attempt to fix the jacobian variable ordering issue

1339089

refactor the normalization of constraints

905d936

adding new examples and refactoring bounds calculation

42f332f

fix issue with bounds creation

8614f3f

finish up a portfolio example

49ffe9a

add example for max

7fc9d88

initial attempts at adding a smooth canon for maximum, what's left?

2e8cd8e

test with more complex example

e5b9352

testing reduction and it seems to work!

a530fba

Merge pull request #2 from cvxgrp/expr2smooth

87ec462

initial attempts at adding a smooth canon for maximum

Merge pull request #1 from cvxpy/ipopt-interface

4544ef7

Ipopt interface prototype

Merge pull request #3 from cvxgrp/working-on-smooth-canons

41bb677

adding more smooth canonicalizers

Merge branch 'cvxpy:master' into master

e75df56

Merge branch 'cvxpy:master' into master

1e8c776

fix interface issues thanks to parth (#5)

ccf69ad

Co-authored-by: William Zijie Zhang <william@gridmatic.com>

update solution statuses thanks to odow (#8)

dcccea0

* update solution statuses thanks to odow * removes unusued solver information --------- Co-authored-by: William Zijie Zhang <william@gridmatic.com>

Adding non-linear example of rocket control (#7)

0a49409

* getting rocket landing example to work * add changes to the jacobian computation --------- Co-authored-by: William Zijie Zhang <william@gridmatic.com>

adding many more example of non-convex functions (#11)

6ea3605

* adding many more example of non-convex functions * making lots of progress on understanding good canonicalizations --------- Co-authored-by: William Zijie Zhang <william@gridmatic.com>

adding some more tests (#12)

b9a7e4d

Co-authored-by: William Zijie Zhang <william@gridmatic.com>

Transurgeon and others added 20 commits February 19, 2026 11:54

Merge branch 'cvxpy:master' into master

a4d0f00

Add differentiation engine section to README

4ef4cd1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Move DerivativeChecker from nlp_solver.py to test helper module (#157)

8b37442

Separates test-only utility from production code by moving it to cvxpy/tests/nlp_tests/derivative_checker.py and updating all 19 test file imports. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

add init files so tests work

0a44098

[Ready for review] Integrating the native COO support in SparseDiff (#…

9138657

…160) * uses new coo handling * added comment * fix pre-commit * fix error message if sample bounds are not set (#161)

Merge upstream/master: sync with cvxpy master

b75b7cb

Includes upstream fixes for quad_form canonicalization, gradient handling, LDL factorization optimization, CI updates, and documentation improvements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Documentation (#164)

e38776b

* uses new coo handling * added comment * fix pre-commit * fix error message if sample bounds are not set * docs for DNLP * remove whitespace * split table up into two

Cache c problem (#167)

8bd8486

* uses new coo handling * added comment * fix pre-commit * fix error message if sample bounds are not set * docs for DNLP * remove whitespace * split table up into two * cache C problem in best of

Merge remote-tracking branch 'upstream/master'

03d9e6c

Merge branch 'master' of https://github.com/cvxgrp/DNLP

99bdad5

remove unnecessary point in dom + clean up C_problem + clean up oracle (

24aac02

#168) * remove unnecessary point in dom + clean up C_problem + clean up oracle * fix william's comments

Transurgeon changed the title ~~Add DIFFENGINE as opt-in canonicalization backend~~ Add DIFFENGINE as canonicalization backend Mar 4, 2026

Transurgeon force-pushed the diffengine-extractor branch from 81e1b7b to 67ba4d8 Compare March 4, 2026 15:01

Transurgeon and others added 5 commits March 4, 2026 11:07

adds changes from experiments, no copy of data, csc output, and more

e02b827

revert back to converting to csc in scipy

30bc8b3

more improvemenets to qp solver interface and more

0031d9a

Transurgeon force-pushed the master branch from 1cbab94 to 6301f44 Compare March 10, 2026 18:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DIFFENGINE as canonicalization backend#172

Add DIFFENGINE as canonicalization backend#172
Transurgeon wants to merge 163 commits intomasterfrom
diffengine-extractor

Transurgeon commented Mar 4, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Transurgeon commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Contribution checklist

Uh oh!

github-actions bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Transurgeon commented Mar 4, 2026 •

edited

Loading

github-actions bot commented Mar 4, 2026 •

edited

Loading