Wrap gradient_free_optimizers (local) #624

gauravmanmode · 2025-07-27T19:30:56Z

PR Description

This PR adds optimizers from gradient_free_optimizers

The following optimizers are now available

Other Changes

Add a new field experimental to AlgoInfo for optimizers that need to skip tests.
Add a new example problem SphereExampleInternalOptimizationProblemWithConverter with converter in internal_optimization_problem.py.
Refactor test_many_algorithms.py, add a PRECISION_LOOKUP dict for algorithm specific precision.

Helper Functions

…izers

codecov · 2025-08-02T19:48:16Z

Codecov Report

❌ Patch coverage is 99.49239% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...agic/optimization/internal_optimization_problem.py	50.00%	1 Missing ⚠️

Files with missing lines	Coverage Δ
src/optimagic/algorithms.py	`88.51% <100.00%> (+0.33%)`	⬆️
src/optimagic/mark.py	`90.00% <ø> (ø)`
src/optimagic/optimization/algorithm.py	`94.97% <100.00%> (+0.02%)`	⬆️
src/optimagic/optimizers/gfo_optimizers.py	`100.00% <100.00%> (ø)`
...agic/optimization/internal_optimization_problem.py	`92.85% <50.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

gauravmanmode · 2025-08-04T14:28:54Z

Hi @janosg
There are a few points

I am failing tests in test_many_algorithms for stochastic algorithms as in the case of all gfo optimizers needs_bounds = True but the tests do not have them.
Writing test for helper functions is a bit difficult as I have exposed the converter. Should I pass arrays only after converting?

janosg · 2025-08-05T06:44:09Z

Hi @gauravmanmode

Is this because we currently use the is_global flag to decide which tests get called wtih bounds? If so, we can simply use the new flags in test_many_algorithms (either adding a new test case for non global optimizers that require bounds or restructuring what weh have).
I don't understand what the problem is and what you mean by "Should I pass arrays only after converting?". Is the problem that SphereExampleInternalOptimizationProblem does not have a converter?

fix doc fixes add tests

gauravmanmode · 2025-08-10T12:39:59Z

Hi @janosg,
Could you please review the changes?

Changes

add a new example problem with converter in internal_optimization_problem

Added functions and converter dealing with dict input.

refactor test_many_algorithms.

( this is minimal and is just for passing tests on this one)

For now, the new tests in test_many_algorithms_new cover all of the tests existing in the original test_many_algorithms_new .
The only failing test is scipy_trust_constr with binding bounds which fails barely . For the purpose of passing this one test, I have set the accuracy of local algorithms to 3 for now.
Why test_with _binding_bounds test does not test with finite bounds ?
Should this be in a separate PR?

gfo

Many algos in GFO do not pass tests with their default values. How should I go about that?
Some algorithms in GFO particularly population based ones do not advance ( n_iterations_performed is low ) if convergence_iter_noimprove is set to something low like < 100. Not sure but some optimizers jump randomly to new positions to explore the search space might be the reason.

src/optimagic/optimizers/gradient_free_optimizers.py

janosg · 2025-08-11T07:43:39Z

src/optimagic/optimizers/gradient_free_optimizers.py

+    stopping_funval: float | None = None
+    """"Stop the optimization if the objective function is less than this value."""
+
+    convergence_iter_noimprove: PositiveInt = 1000  # need to set high


Should we set this to None or a really high value instead? Is there another convercence criterion we could set to a non-None value instead? We don't want all optimizers just to run until max_iter but we also don't want pre-mature stopping of course.

I am a bit confused about this.
Most of the time convergence_iter_noimprove behaves as stopping_maxiter. The other convergence criteria dont seem to be respected. Even after a > 100000 iterations the algorithm does not converge to a good solution. I might be missing something but hope to get a solution here.

Thanks for creating the issue over there.

Since convergence_iter_noimprove is set to a large value, it will look for a change in a large no. of iterartions and convergence_ftol_abs and convergence_ftol_rel will not have effect.

src/optimagic/optimizers/gradient_free_optimizers.py

tests/optimagic/optimization/test_many_algorithms_new.py

…st_gfo_optimizers

…okup dict for failing tests, move nag_dfols test to nag_optimizers, move test_pygmo_optimizers

…ion, tests

gauravmanmode · 2025-08-27T13:37:01Z

Hi @janosg, The tests are passing, could you review this as well?

janosg · 2025-08-28T06:15:55Z

@gauravmanmode can you fix the merge conflicts?

…mizers

gauravmanmode · 2025-08-28T08:16:59Z

@janosg , I've fixed the merge conflicts.

janosg

Very nice. Just minor comments!

janosg · 2025-08-30T07:13:11Z

docs/source/algorithms.md

+```{eval-rst}
+.. dropdown:: Common options across all optimizers
+
+  .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOCommonOptions


I probably missed this last time: If possible I would prefer not to expose the common options in the documentation and instead document the inherited attributes for each optimizer. I think it is possible with something like this:

.. autoclass:: SomeGFOOptimizer :members: :inherited-members: GFOCommonOptions

Will work on this.

Thanks for the code snippet. I achieved similar with

.. autoclass:: SomeGFOOptimizer :members: :inherited-members: Algorithm, object :member-order: bysource

The members of classes listed in the argument to :inherited-members: are excluded from the automatic documentation.
:member-order: bysource will list base class members first.

Looks good to me. You can decide on the implementation. The goal is that the final documentation of each algorithm looks like it would look if we did not use inheritance from common options.

janosg · 2025-08-30T07:14:27Z

src/optimagic/optimizers/gfo_optimizers.py

@@ -108,6 +109,482 @@ class GFOCommonOptions:
    seed: int | None = None
    """Random seed for reproducibility."""

+    rand_rest_p: NonNegativeFloat = 0


Should this be a ProbabilityFloat? Similar question for other options below.

Yes, I missed that, p_accept is also ProbabilityFloat.

janosg · 2025-08-30T07:15:37Z

src/optimagic/optimizers/gfo_optimizers.py

+    Hill climbing is a local search algorithm suited for exploring combinatorial search
+    spaces.
+
+    It starts at an initial point, which is often chosen randomly and continues to move


I would remove "which is often chosen randomly". There are different ways to get start parameters and many include domain knowledge.

the implemenation in GFO is that it runs a initialization step where it queries n_init points (chosen from grid randomly or vertices of the grid) and starts from the best one. should i mention this.

Yes, that would be better. Currently it sounds like one random init point.

src/optimagic/optimizers/gfo_optimizers.py

review-notebook-app · 2025-08-31T11:07:00Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

gauravmanmode added 6 commits July 27, 2025 17:45

initial wrapper over hill_climbing optimizer

60a7185

Merge remote-tracking branch 'upstream/main' into gradient_free_optim…

c101bd8

…izers

lazy_loading import and add docstrings for class and helper func

fdc8097

fix path in docs

a37771a

add base class for common options , add pso optimizer

d190062

grid points can be pytree

67d9904

Merge branch 'main' into gradient_free_optimizers

cfe87f8

gauravmanmode marked this pull request as ready for review August 4, 2025 14:21

gauravmanmode added 2 commits August 4, 2025 20:34

add test, n_init option

e5e5175

add hillclimbing derivatives and docs

23dc21d

gauravmanmode added 10 commits August 5, 2025 17:30

add tests

27d0b1b

remove pso

73946de

refactor test_many_algorithms new

9948a8b

tune gfo algorithms temp to pass tests

90694b8

add sphereinternaloptimizationexample with converter

a1cd6cd

fix doc fixes add tests

pytree tests for gfo

c2958fd

run new test_many comment old test_many

17bce2e

mypy override gfo

d341b47

tune gfo to pass tests

312ded8

set accuracy of local from 4 to 3 temp

45f86e6

janosg reviewed Aug 11, 2025

View reviewed changes

tests/optimagic/optimization/test_many_algorithms_new.py Outdated Show resolved Hide resolved

gauravmanmode added 4 commits August 12, 2025 07:24

rename to gfo_optimizers, rename simplex parameters, add to ignore mypy,

a95a826

pass common_options as first argument to reduce clutter, rename to te…

deae6c0

…st_gfo_optimizers

negate fun value

fb3ba70

test_many_algorithms add finite case for binding bounds, precision lo…

d55d8e0

…okup dict for failing tests, move nag_dfols test to nag_optimizers, move test_pygmo_optimizers

gauravmanmode added 7 commits August 13, 2025 12:47

to default values

290c88f

tree bounds in sphereexamplewithconverter

3bdde60

pass self directly in common_options

0512957

add_other_algos

3cfc2c1

only local opts, rename to extra_strat_points, and add warm_start opt…

f9405ab

…ion, tests

docs, add experimental field

6eceedb

merge from gfo_pop, tests, renaming

38f2db0

Merge branch 'main' into gradient_free_optimizers

8d76a16

gauravmanmode added 4 commits August 28, 2025 13:17

Wrap gradient_free_optimizers (pop_based) (#636)

a013de4

lower stopping maxtime to 1 sec

16680b9

pre-commit fixes

b287027

Merge remote-tracking branch 'upstream/main' into gradient_free_opti…

3c6f6b1

…mizers

Merge branch 'main' into gradient_free_optimizers

bef8ecd

janosg reviewed Aug 30, 2025

View reviewed changes

janosg mentioned this pull request Aug 30, 2025

Add optimizers from PySwarms #639

Open

gauravmanmode added 5 commits August 31, 2025 16:36

update type hints

68f25d0

dont expose commonoptions

daa9dbe

update docstring hillclimbing , stochastchillclimbing

5ff94d9

fix in sphereexmaplewithconverter

efe6dc1

self suff examples

451de8c

Merge branch 'main' into gradient_free_optimizers

b0b70aa

Wrap gradient_free_optimizers (local) #624

Are you sure you want to change the base?

Wrap gradient_free_optimizers (local) #624

Uh oh!

Conversation

gauravmanmode commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Description

Other Changes

Helper Functions

Uh oh!

codecov bot commented Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gauravmanmode commented Aug 4, 2025

Uh oh!

janosg commented Aug 5, 2025

Uh oh!

gauravmanmode commented Aug 10, 2025

Changes

add a new example problem with converter in internal_optimization_problem

refactor test_many_algorithms.

gfo

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gauravmanmode Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gauravmanmode commented Aug 27, 2025

Uh oh!

janosg commented Aug 28, 2025

Uh oh!

gauravmanmode commented Aug 28, 2025

Uh oh!

janosg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gauravmanmode Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

review-notebook-app bot commented Aug 31, 2025

Uh oh!

Uh oh!

gauravmanmode commented Jul 27, 2025 •

edited

Loading

codecov bot commented Aug 2, 2025 •

edited

Loading

gauravmanmode Aug 12, 2025 •

edited

Loading

gauravmanmode Aug 30, 2025 •

edited

Loading